requests Python Library Primer

Install

You can install using pip/easy_install/buildout:

./bin/pip install requests

Basics

Request a url, get the results back:

import requests
resp = requests.get('https://cdn.api.twitter.com/1/urls/count.json?url=https://www.cnn.com')
resp.content
resp.json()


Basic Auth

Many APIs you interact with use simple basic authentication:

endpoint = 'https://foo.bar'
resp = requests.get(endpoint)
resp.status_code
resp = requests.get(endpoint, auth=('user', 'pass'))
resp.status_code

With LXML for scraping

Install lxml the same way you did requests:

./bin/pip install lxml
./bin/pip install cssselectUsing lxml to parse response:

from lxml import html
import requests
resp = requests.get('http://www.cnn.com')
dom = html.fromstring(resp.content)
html.tostring(dom.cssselect('h2')[0])

Get meta tags:

print([html.tostring(e) for e in dom.cssselect('meta')])