Skip to content

mYQL is a Python wrapper of the Yahoo! Query Language a.k.a YQL (Finances, ...)

License

Notifications You must be signed in to change notification settings

josuebrunel/myql

Repository files navigation

Build Status Documentation Status Code Health PyPI PyPI PyPI PyPI PyPI Coverage Status PyPI

mYQL is a Python wrapper of the Yahoo Query Language.

Yahoo! Query Language Documentation and Support

Features

  • Simple YQL Query
  • Authenticated YQL Query ( OAuth )
  • StockScraper
  • YQL Open Table (Classes and Metaclasses) Generator
  • Response prettyfier

Installation

$ pip install myql

Quick Start

It's important to know that response is a just requests.models.Response object. Yes indeed, mYQL uses requests 😄

By default, you have access to the community tables. If for whatsoever reason you would like to not have access to those tables

>>> import myql
>>> yql = myql.MYQL(community=False)

####Changing response format (xml or json)

The response format is by default json.

>>> import myql
>>> from myql.utils import pretty_json, pretty_xml
>>> yql = myql.MYQL(format='xml', community=True)
>>> resp = yql.raw_query('select name, woeid from geo.states where place="Congo"')
>>> print(pretty_xml(resp.content))
<?xml version="1.0" encoding="utf-8"?>
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="11" yahoo:created="2015-06-07T11:56:11Z" yahoo:lang="en-US">
    <results>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Cuvette-Ouest Department</name>
            <woeid>55998384</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Cuvette Department</name>
            <woeid>2344968</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Plateaux District</name>
            <woeid>2344973</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Sangha</name>
            <woeid>2344974</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Lekoumou</name>
            <woeid>2344970</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Pool Department</name>
            <woeid>2344975</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Likouala Department</name>
            <woeid>2344971</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Niari Department</name>
            <woeid>2344972</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Brazzaville</name>
            <woeid>2344976</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Bouenza Department</name>
            <woeid>2344967</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Kouilou</name>
            <woeid>2344969</woeid>
        </place>
    </results>
</query>
<!-- total: 33 -->
<!-- pprd1-node1003-lh3.manhattan.bf1.yahoo.com -->

>>> resp = yql.raw_query('select name, woeid from geo.states where place="Congo"', format='json')
>>> print(pretty_json(resp.content))
{
    "query": {
        "count": 11, 
        "created": "2015-06-07T11:58:20Z", 
        "lang": "en-US", 
        "results": {
            "place": [
                {
                    "name": "Cuvette-Ouest Department", 
                    "woeid": "55998384"
                }, 
                {
                    "name": "Cuvette Department", 
                    "woeid": "2344968"
                }, 
                {
                    "name": "Plateaux District", 
                    "woeid": "2344973"
                }, 
                {
                    "name": "Sangha", 
                    "woeid": "2344974"
                }, 
                {
                    "name": "Lekoumou", 
                    "woeid": "2344970"
                }, 
                {
                    "name": "Pool Department", 
                    "woeid": "2344975"
                }, 
                {
                    "name": "Likouala Department", 
                    "woeid": "2344971"
                }, 
                {
                    "name": "Niari Department", 
                    "woeid": "2344972"
                }, 
                {
                    "name": "Brazzaville", 
                    "woeid": "2344976"
                }, 
                {
                    "name": "Bouenza Department", 
                    "woeid": "2344967"
                }, 
                {
                    "name": "Kouilou", 
                    "woeid": "2344969"
                }
            ]
        }
    }
}

>>>

Methods

####use(yql_table_url,name=yql_table_name) Maps a table name to the URL of an Open Data Table.

>>> yql.use('http://www.josuebrunel.org//users.xml', name='myusers') 

####desc(tablename) Returns table description

>>> response = yql.desc('weather.forecast')
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1, 
        "created": "2015-06-07T12:00:27Z", 
        "lang": "en-US", 
        "results": {
            "table": {
                "hash": "aae78b1462a6a8fbc748aec4cf292767", 
                "meta": {
                    "author": "Yahoo! Inc", 
                    "description": "Weather forecast table", 
                    "documentationURL": "http://developer.yahoo.com/weather/", 
                    "sampleQuery": "select * from weather.forecast where woeid=2502265"
                }, 
                "name": "weather.forecast", 
                "request": {
                    "select": [
                        {
                            "key": [
                                {
                                    "name": "location", 
                                    "required": "true", 
                                    "type": "xs:string"
                                }, 
                                {
                                    "name": "u", 
                                    "type": "xs:string"
                                }
                            ]
                        }, 
                        {
                            "key": [
                                {
                                    "name": "woeid", 
                                    "required": "true", 
                                    "type": "xs:string"
                                }, 
                                {
                                    "name": "u", 
                                    "type": "xs:string"
                                }
                            ]
                        }
                    ]
                }, 
                "security": "ANY"
            }
        }
    }
}

>>>

####raw_query(query)

Allows you to directly type your query

>>> response = yql.raw_query("select * from geo.countries where place='North America'")
>>> # deal with the response

####select(table, fields, limit).where(filters, ...)

NB : A simple select doesn't return any data. Use GET instead.

>>> response = yql.select('geo.countries', ['name', 'code', 'woeid']).where(['name', '=', 'Canada'])
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1, 
        "created": "2015-06-07T12:10:39Z", 
        "lang": "en-US", 
        "results": {
            "place": {
                "name": "Canada", 
                "woeid": "23424775"
            }
        }
    }
}

>>> ...
>>> response = yql.select('geo.countries', ['name', 'woeid'], 2).where(['place', 'in', ('Africa', 'Europe')])
>>> from myql.utils import dump
>>> dump(response)
{
    "query": {
        "count": 2, 
        "created": "2015-06-07T12:27:04Z", 
        "lang": "en-US", 
        "results": {
            "place": [
                {
                    "name": "Algeria", 
                    "woeid": "23424740"
                }, 
                {
                    "name": "Angola", 
                    "woeid": "23424745"
                }
            ]
        }
    }
}

>>>

####get(table, fields, limit) Same as SELECT, but instead returns data.

REMINDER : Some tables require a where clause, therefore GET won't work on those tables, use select(...).where(...) instead .

>>> from myql.utils import dump
>>> response = yql.get('geo.countries', ['name', 'woeid'], 1)
>>> dump(response)
{
    "query": {
        "count": 1, 
        "created": "2015-06-07T12:29:01Z", 
        "lang": "en-US", 
        "results": {
            "place": {
                "name": "Sao Tome and Principe", 
                "woeid": "23424966"
            }
        }
    }
}

>>>

####insert(table, (field1, field2, ..., fieldN),(value1, value2, ..., valueN)) Insert values into a table. Arguments 2 and 3 may be tuples or list.

>>> from myql.utils import pretty_json
>>> response = yql.insert('yql.storage.admin',('value',),('http://josuebrunel.org',))
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1,
        "created": "2015-05-14T13:25:56Z",
        "lang": "en-US",
        "results": {
            "inserted": {
                "execute": "store://KkkC5xDw4v32IcWWSQ4YRe",
                "select": "store://Zc5LHXcmYM7XBfSbo9tzFL",
                "update": "store://Rqb5fbQyDvrfHJiClWnZ6q"
            }
        }
    }
}

####update(table,[field1, ..., fieldN],[value1, ..., ...valueN]).where(filters, ...) Update fields values. This method is always followed by where(). Arguments 2 and 3 may be tuples or list.

>>> from myql.utils import pretty_json
>>> response = yql.update('yql.storage',('value',),('https://josuebrunel.org',)).where(['name','=','store://Rqb5fbQyDvrfHJiClWnZ6q'])
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1,
        "created": "2015-05-14T13:32:52Z",
        "lang": "en-US",
        "results": {
            "success": "Updated store://KkkC5xDw4v32IcWWSQ4YRe"
        }
    }
}

####delete(table).where(filters, ...) Delete records

>>> from myql.utils import pretty_json
>>> response = self.yql.delete('yql.storage').where(['name','=','store://Rqb5fbQyDvrfHJiClWnZ6q'])
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1,
        "created": "2015-05-14T13:38:28Z",
        "lang": "en-US",
        "results": {
            "success": "store://Rqb5fbQyDvrfHJiClWnZ6q deleted"
        }
    }
}

####Using OAuth

mYQL comes with yahoo_oauth, which is an OAuth library for Yahoo! APIs.

>>> from yahoo_oauth import OAuth1
>>> oauth = OAuth1(None, None, from_file='credentials.json') # only consumer_key and consumer_secret are required.
>>> from myql import MYQL
>>> yql = MYQL(format='xml', oauth=oauth)
>>> response = yql.get_guid('josue_brunel') # Deal with the response

Stocks Scraper

The full documentation on StockScraper is here

Release Notes

1.2.6

1.2.5

  • camelCase dropped for underscore
  • Support for substitution variable i.e @myvar
  • Support of Remote Filters
  • Support of Post Query Filters
1.2.4

  • Weather module added
  • StockScraper now under Finance namespace
1.2.3

  • Fixed issue related to date in StockRetriver.get_historical_info #107
  • Fixed issue with IN condition in where clause #106
  • Fix definition of raw_input for python3 #105
  • Yahoo-OAuth included as main oauth library #112
1.2.2

  • Python3 support OK #71
  • PyPy/PyPy3 support OK
  • Fixed issue with IN condition in where clause
  • Fixed issue when passing an empty list/tuple ([]/()) in a where clause besides first argument
  • Import of StockParser from Gurchet Rai OK #68
  • Insert, Update, Delete methods added #67
  • Dummy try/except removed from main module
  • Fixed Invalid OAuth Signature when using a refreshed token #64
  • Fixed misused of MYQL.use(...) #76
  • Fixed format issue #82
  • Added useful functions in utils #81
  • Default access to community tables
  • Response prettyfier : pretty_json, pretty_xml
v 1.2.1

  • Multiple requests while using OAuth fixed
1.2.0

  • OpenTable classes
  • Access to resources requiring authentication
0.5.6

  • fetch data
  • access to community data
  • select data format (xml/json)
  • change data source
  • filter data
  • fix handling of default response format on the fly
  • fix limit on select(...).where(...) when no limit value is passed
  • fix limit on get(...)