URL checking with Python

Use the right tool for the right tasks, dealing with HTTP was never easier than with the Python Requests library.

One pattern I keep using over and over is reading some data from a database and do something with it. Combined with Requests it is trivial to create a link checker.

By using the DictCursor of MySQLdb the data becomes accesible via a Python dictionary which is easy to read. There is no particular reason that I picked MySQLdb other than at the time it was the only MySQL driver for Python. There are others now, like ourSQL and Connector/Python but I have no experience with them.

Easiest way to install stuff is with pip,

pip install MySQLdb

pip install requests

If for some reason pip is not installed then Google for the preferred way to install it on your platform.

Without further ado here is the code:

 

# python 2.x script to fetch urls and verify them
import MySQLdb
import MySQLdb.cursors
import requests
db=MySQLdb.connect(user="user", passwd="pass",db="database", cursorclass=MySQLdb.cursors.DictCursor)
c = db.cursor()
q = "SELECT url FROM table"
c.execute(q)
result = c.fetchall()

for r in result:
    try:
        rq = requests.get( r
['url'])
        if rq.status_code==404:
            print "%-80s " % r['url'], " ", i, rq.status_code

    except Exception, e:
        print e