July 17, 2012

Output Data as an HTML File with Python

Reviewed by Miriam Posner and Jim Clifford

Note: You may find it easier to complete this lesson if you have already completed the previous lesson in this series.

Lesson Goals

This lesson takes the frequency pairs created in Counting Frequencies and outputs them to an HTML file.

Here you will learn how to output data as an HTML file using Python. You will also learn about string formatting. The final result is an HTML file that shows the keywords found in the original source in order of descending frequency, along with the number of times that each keyword appears.

Files Needed For This Lesson

• obo.py

If you do not have these files from the previous lesson, you can download programming-historian-6, a zip file from the previous lesson

Building an HTML wrapper

In the previous lesson, you learned how to embed the message “Hello World!” in HTML tags, write the result to a file and open it automatically in the browser. A program that puts formatting codes around something so that it can be used by another program is sometimes called a wrapper. What we’re going to do now is develop an HTML wrapper for the output of our code that computes word frequencies. We’re also going to add some helpful, dynamic metadata to supplement the frequency data collected in Counting Frequencies.

When you’re working with digital data, it is a good idea to incorporate metadata into your own files whenever possible. We will now develop a few basic strategies for making our data files self-documenting. In our wrapper, we want to include dynamic information about the file, such as the time and date it was created, as well as an HTML title that is relevant to the file. In this case we could just give it a name ourselves, but when we start working with multiple files, automatically creating self-documenting files will save a lot of time, so we’ll practice now. And for that, we’ll have to learn to take advantage of a few more powerful string formatting options.

Python string formatting

Python includes a special formatting operator that allows you to insert one string into another one. It is represented by a percent sign followed by an “s”. Open a Python shell and try the following examples.

frame = 'This fruit is a %s'
print(frame)
-> This fruit is a %s

print(frame % 'banana')
-> This fruit is a banana

print(frame % 'pear')
-> This fruit is a pear


There is also a form which allows you to interpolate a list of strings into another one.

frame2 = 'These are %s, those are %s'
print(frame2)
-> These are %s, those are %s

print(frame2 % ('bananas', 'pears'))
-> These are bananas, those are pears


In these examples, a %s in one string indicates that another string is going to be embedded at that point. There are a range of other string formatting codes, most of which allow you to embed numbers in strings in various formats, like %i for integer (eg. 1, 2, 3), %f for floating-point decimal (eg. 3.023, 4.59, 1.0), and so on. Using this method we can input information that is unique to the file.

Self-documenting data file

Let’s bundle some of the code that we’ve already written into functions. One of these will take a URL and return a string of lowercase text from the web page. Copy this code into the obo.py module.

# Given a URL, return string of lowercase text from page.

def webPageToText(url):
import urllib2
response = urllib2.urlopen(url)
text = stripTags(html).lower()
return text


We’re also going to want a function that takes a string of any sort and makes it the body of an HTML file which is opened automatically in Firefox. This function should include some basic metadata, like the time and date that it was created and the name of the program that created it. Study the following code carefully, then copy it into the obo.py module.

Mac Instructions

If you are using a Mac, make sure you include the proper file path in the filename variable on the 2nd last line to reflect where you’re saving your files.

# Given name of calling program, a url and a string to wrap,
# output string in html body with basic metadata and open in Firefox tab.

def wrapStringInHTMLMac(program, url, body):
import datetime
from webbrowser import open_new_tab

now = datetime.datetime.today().strftime("%Y%m%d-%H%M%S")
filename = program + '.html'
f = open(filename,'w')

wrapper = """<html>
<title>%s output - %s</title>
<body><p>URL: <a href=\"%s\">%s</a></p><p>%s</p></body>
</html>"""

whole = wrapper % (program, now, url, url, body)
f.write(whole)
f.close()

#Change the filepath variable below to match the location of your directory

open_new_tab(filename)


Windows Instructions

# Given name of calling program, a url and a string to wrap,
# output string in html body with basic metadata
# and open in Firefox tab.

def wrapStringInHTMLWindows(program, url, body):
import datetime
from webbrowser import open_new_tab

now = datetime.datetime.today().strftime("%Y%m%d-%H%M%S")

filename = program + '.html'
f = open(filename,'w')

wrapper = """<html>
<title>%s output - %s</title>
<body><p>URL: <a href=\"%s\">%s</a></p><p>%s</p></body>
</html>"""

whole = wrapper % (program, now, url, url, body)
f.write(whole)
f.close()

open_new_tab(filename)


***

Note that this function makes use of the string formatting operator about which we just learned. If you are still having trouble with this idea, take a look at the HTML file that opened in your new Firefox tab and you should see how this worked. If you’re still stuck, take a look at the

URL: http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33


in the HTML file and trace back how the program knew to put the URL value there.

The function also calls the Python datetime library to determine the current time and date. Like the string formatting operator %s, this library uses the % as replacements for values. In this case, the %Y %m %d %H %M %S represents year, month, date, hour, minute and second respectively. Unlike the %s, the program will determine the value of these variables for you using your computer’s clock. It is important to recognize this difference.

This date metadata, along with the name of the program that called the function, is stored in the HTML title tag. The HTML file that is created has the same name as the Python program that creates it, but with a .html extension rather than a .py one.

Putting it all together

Now we can create another version of our program to compute frequencies. Instead of sending its output to a text file or an output window, it sends the output to an HTML file which is opened in a new Firefox tab. From there, the program’s output can be added easily as bibliographic entries to Zotero. Type or copy the following code into your text editor, save it as html-to-freq-3.py and execute it, to confirm that it works as expected.

Use either obo.wrapStringInHTMLMac() or obo.wrapStringInHTMLWindows() as appropriate for your system.

# html-to-freq-3.py
import obo

# create sorted dictionary of word-frequency pairs
url = 'http://www.oldbaileyonline.org/browse.jsp?id=t17800628-33&div=t17800628-33'
text = obo.webPageToText(url)
fullwordlist = obo.stripNonAlphaNum(text)
wordlist = obo.removeStopwords(fullwordlist, obo.stopwords)
dictionary = obo.wordListToFreqDict(wordlist)
sorteddict = obo.sortFreqDict(dictionary)

# compile dictionary into string and wrap with HTML
outstring = ""
for s in sorteddict:
outstring += str(s)
outstring += "<br />"
obo.wrapStringInHTMLMac("html-to-freq-3", url, outstring)


Note that we interspersed our word-frequency pairs with the HTML break tag <br\>, which acts as a newline. If all went well, you should see the same word frequencies that you computed in the last section, this time in your browser window.

• Lutz, Learning Python
• Re-read and review Chs. 1-17

Code Syncing

To follow along with future lessons it is important that you have the right files and programs in your “programming-historian” directory. At the end of each chapter you can download the “programming-historian” zip file to make sure you have the correct code. If you are following along with the Mac / Linux version you may have to open the obo.py file and change “file:///Users/username/Desktop/programming-historian/” to the path to the directory on your own computer.

Note: You are now prepared to move on to the next lesson in this series.