Web Page Management Software
John Hurst
Version 1.3.6
20160221:165516
Abstract
This document defines and describes the suite of programs
used to create my web page environment on the range of
machines that I use.
Table of Contents
1. Introduction
This document describes the files used to manage delivery of my
personal web pages, and those that I manage for other
organisations. The general form of web page delivery is a) a
source file written in XML, b) a translation file written in
XSLT, and c) the program described here, a python cgi script
that calls the appropriate translator on the source file, and
delivers the result. It also handles straight HTML, as well as
providing some debug and other maintenance options.
The program is invoked by commands in the .htaccess file
associated with each web directory. Different .htaccess
files can be used for different directories. If none exist in a
given directory, the directory path is searched towards the root
until one is found.
The XSLT files used can be specified either in the
.htaccess file (default), or in the source XML file,
through an explicit xml-stylesheet command. If a stylesheet
XSLT file is specified, it overrides the default
.htaccess one.
Permission is given to reuse this document, provided that the
source is acknowledged, and that any changes are noted in the
documentation.
The document is in the form of a literate program, and
generates all files necessary to maintain the working
environment, including a Makefile.
2. Literate Data
<edit warning 2.1> =#
# DO NOT EDIT this file!
# see $HOME/Computers/Sources/Web/web.xlp instead
# this also gives further explanation of the program logic
#
This message flags the fact that the source code is a derived
document, and should not be directly edited..
3. The Main Program index.py
"index.py" 3.1 =#!/usr/bin/python
<edit warning 2.1>
version="
<current version .1>"
# This script processes all my web page XML files
# It requires apache to be configured:
# a) to allow python files (this file) to run as cgi in user directories
# b) to add a handler for XML files that calls this program
# c) to pass as cgi parm the XSLT file that translates the XML file
# These are done in a .htaccess file for each directory (and its
# subdirectories) that require XML processing with a particular
# XSLT stylesheet.
#
# The script relies upon picking up the required file and its XSLT file
# from a) the REDIRECT environment variables, and b) the script parameter,
# respectively.
#
#
Start with a bit of explanatory comment in case direct access
to this literate program is unavailable.
The interpreter required varies according to the target server.
This detail is captured by the <Makefile 9.2> script, although not all systems have yet been encoded into the Makefile script.
"index.py" 3.2 =import cgi ; import cgitb ; cgitb.enable()
import cgi
import commands
import datetime
import os, os.path
import re
import sys
from subprocess import PIPE,Popen
import time
import urllib2
import urlparse
import xml.dom.minidom
Gather together all the module and library imports needed for
this program.
"index.py" 3.3 =now=datetime.datetime.now()
tsstring=now.strftime("%Y%m%d:%H%M")
todayStr=now.strftime("%d %b %Y")
htmlmod=xmlmod=0
Start processing. Get a timestamp for recording key events in
the log. Set the modification times to year dot.
"index.py" 3.4 =
Define a useful debug routine.
"index.py" 3.5 =def debug(loc,msg):
if debugFlag: print msg
"index.py" 3.6 =
This fragment simply outputs the required header for flagging
the generated content as HTML.
"index.py" 3.7 =
These two fragments explore the incoming parameters in order to
find out what file is to be processed.
"index.py" 3.8 =
<check for abbreviated URL 3.19>
<make file and dir absolute 3.20>
<check for HTML request 3.21,3.22>
<get default XSLT file 3.24>
<scan for locally defined XSLT file 3.25>
<determine xslt file 3.27>
<update counter 3.28,3.29,3.30,3.31>
if
debugFlag:
print "\n<p>\n"
print "%s: server = %s<br/>" % (tsstring,server)
print "%s: host = %s<br/>" % (tsstring,host)
print "%s: dir = %s<br/>" % (tsstring,dir)
print "%s: requestedFile = %s<br/>" % (tsstring,
requestedFile)
print "%s: relcwd = %s<br/>" % (tsstring,relcwd)
print "%s: relfile = %s<br/>" % (tsstring,relfile)
print "%s: counter = %s<br/>" % (tsstring,counterName)
print "%s: alreadyHTML = %s<br/>" % (tsstring,alreadyHTML)
print "%s: cachedHTML = %s<br/>" % (tsstring,cachedHTML)
print "%s: os.environ = %s\n</p>\n" % (tsstring,repr(os.environ))
print "%s: docRoot = %s\n</p>\n" % (tsstring,docRoot)
<process file 3.32,3.33>
now=datetime.datetime.now()
tsstring=now.strftime("%Y%m%d:%H%M")
#sys.stderr.write("%s: [client %s] request satisfied\n" % (tsstring,remoteAdr))
At this point, processing is complete, and the program falls
through to exit.
3.1 Define Various String Patterns
<define various string patterns 3.9> =# - to extract directory and filename from request
filepat=re.compile('/~ajh/?(.*)/([^/]*)$')
filename='index.xml'
# - to detect stylesheet request (optional)
stylesheet=re.compile('<\?xml-stylesheet.*href="(.*)"')
# - to terminate file scanning
doctype=re.compile('<!DOCTYPE')
# to check for missing htmls
htmlpat=re.compile('(.*)\.html$')
# to check for xslspecification in htaccess
xslspec=re.compile('.*?xslfile=(.*)&')
3.2 Define Global Variables and Constants
<define global variables and constants 3.10> =
returnXML is set True when the display of the
raw untranslated XML is required.
convertXML is set True when a converted copy of
the translated XML is required to be saved.
alreadyHTML is set True when the incoming file
to be rendered is already in HTML and does not require
conversion.
cachedHTML is set True when the incoming file
to be rendered has been cached in the HTMLS directory,
and does not require conversion.
<define global variables and constants 3.11> =# begin 1 - define global variables and constants
# BASE is the path to the web base directory - with no trailing slash!
if system==MacOSX:
HOME="/home/ajh/"
BASE="/home/ajh/www"
host=re.sub('dyn-130-194-\d+-\d+','dyn',host)
PRIVATE="/home/ajh/local/"+server
elif system==Solaris:
HOME="/u/staff1/ajh"
BASE="/u/web/homes/ajh"
PRIVATE=BASE+"/local/"+server
elif system==Ubuntu and server=='cahurst.org':
HOME="/home/ajh"
BASE="/home/ajh/www/personal/cahurst"
PRIVATE="/home/ajh/local/"+server
elif system==Ubuntu:
if debugFlag: print "docRoot=%s" % (docRoot)
if docRoot == '/var/www/cerg':
HOME="/var/www/cerg/"
BASE="/var/www/cerg/"
PRIVATE="/var/www/cerg/local/"+server
elif docRoot == '/home/ajh/public_html/parish/GWICC':
HOME="/home/ajh/public_html/parish/GWICC"
BASE=HOME
PRIVATE="/home/ajh/local/"+server+"/parish/GWICC"
elif docRoot == '/home/ajh/public_html/personal/cahurst':
HOME="/home/ajh/public_html/personal/cahurst"
BASE=HOME
PRIVATE="/home/ajh/local/"+server+"/personal/cahurst"
else:
HOME="/home/ajh/"
BASE="/home/ajh/www"
PRIVATE="/home/ajh/local/"+server
COUNTERS=PRIVATE+"/counters/"
HTMLS=PRIVATE+"/htmls/"
WEBDIR="file://"+BASE
# end 1 - define global variables and constants
is set to the path to the web root
directory on this server. It should not have a
trailing slash!
has its usual Unix meaning.
is set to the path to a working
directory on this particular server that is used to store
accounting and audit information about this particular access.
The path includes a specific reference to the server hostname
to uniquely distinguish it.
is the path to a local copy of html
versions of the files. These are cached versions, and some
mechanism to age and delete needs to be identified. If the
corresponding XML file is older than the HTML file found in
this subdirectory, the HTML version is used.
<define global variables and constants 3.12> =# begin 2 - define global variables and constants
# define the XSLTPROC
if system==MacOSX:
XSLTPROC="/usr/bin/xsltproc"
elif system==Solaris:
XSLTPROC="/usr/monash/bin/xsltproc"
elif system==Linux:
XSLTPROC="/usr/bin/xsltproc"
elif system==Ubuntu:
XSLTPROC="/usr/bin/xsltproc"
# end 2 - define global variables and constants
XSLTPROC is the path to the xsltproc processor.
Without this processor, this entire script (as far as XML
files are concerned) is meaningless!
3.3 Handle JPEGs
From version 1.2.0 onwards, this code implements a form of
caching for jpg files. A local check for the request file is
made, and if it is not found, an attempt to retrieve it from
the dimboola server is made. If that is not successful, the
file is reported not found. If it is successful, the file is
saved locally. No attempt is made to age files out of the
cache.
<check for and handle jpegs 3.13> =#sys.stderr.write("Just a check version %s\n" % (version))
cachetime=60*60*24*7 # one week
# check for jpgs
if os.environ.has_key('REQUEST_URI'):
uri=os.environ['REQUEST_URI']
(scheme,netloc,path,parms,query,fragment)=urlparse.urlparse(uri)
#sys.stderr.write("path=%s\n" % path)
filename=re.sub('/~ajh/','/home/ajh/www/',path)
(root,ext)=os.path.splitext(filename)
ext=ext.lower()
if ext=='.jpg':
basedir=os.path.dirname(filename)
if os.path.exists(filename):
sys.stderr.write("Got file %s locally\n" % filename)
f=open(filename,'r').read()
else:
<get jpg file from remote server 3.14,3.15>
print "Content-Type: image/jpeg\n"
print f # display the image
sys.exit(0)
else:
pass
This code checks to see if the request is for a jpg image
file. These are cached, and if not present, are retrieved
from the master jpg server for my jpeg images. This is still
a bit experimental. The server URL is
dimboola.infotech.monash.edu.au/~ajh/Pictures.
It requires that the .htaccess file be modified to
refer .jpg requests to this cgi script.
3.3.1 Get JPG File from Remote Server
<get jpg file from remote server 3.14> =newurl="http://ajh.id.au%s" % path
sys.stderr.write("using url %s\n" % newurl)
urlobj=urllib2.urlopen(newurl)
f=urlobj.read()
modtimestr=urlobj.info()['Last-Modified']
modtime=time.strptime(modtimestr,"%a, %d %b %Y %H:%M:%S %Z")
Generate the URL of the corresponding remote JPG file, and
issue read request. By using the
urllib2
library, we also get the modification time, which we parse
in order to set the correct modification time on the locally
cached copy.
<get jpg file from remote server 3.15> =try:
fc=open(filename,'w')
fc.write(f)
fc.close()
#touch filename -mt time.strftime("%Y%m%d%H%M.%S")
mtime=time.mktime(modtime)
imtime=int(mtime)
nowtime=time.localtime()
currtime=int(time.mktime(nowtime)) # local
os.utime(filename,(currtime,imtime))
#sys.stderr.write("%s: cached %s\n" % (tsstring,filename))
except IOError,OSError:
#errmsg=os.strerror(errcode)
sys.stderr.write("%s: Cannot write cache file %s\n" % (tsstring,filename))
Now try to cache a local copy. This can fail for several
reasons, the main one being that the permissions in the
local directory are likely to be against (write) access by
the www user. More work is required to make this a
bit more robust.
Note that we set the pair (access time, modification time)
on the local file to be the current time and remote file
modification time respectively. This ensures that attempts
to synchronize the two file systems will see this file as
the same file as the remote file, and not attempt to update
one or the other (thus leading to spurious modification
times).
3.4 Collect HTTP Request
<collect HTTP request 3.16> =# collect the original parameters from the redirect (if there is one!)
if os.environ.has_key('REDIRECT_QUERY_STRING'):
<handle redirect query string 3.17>
else:
form={}
requestedFile=""
{Note 3.16.1}
remoteAdr=''
if os.environ.has_key('REMOTE_ADDR'):
remoteAdr=os.environ['REMOTE_ADDR']
if
debugFlag:
print "<p>%s: (server,host)=(%s,%s)<br/>\n" % (tsstring,server,host)
print "%s: (system,PRIVATE)=(%s,%s)</p>\n" % (tsstring,system,PRIVATE)
print "%s: (BASE,HOME,PRIVATE)=(%s,%s,%s)</p>\n" % (tsstring,BASE,HOME,PRIVATE)
- {Note 3.16.1}
- initialize the filename of the file to be rendered. Most of the work in computing the value of this variable is done in <get filename from redirect 3.18>
When this script is called, it has gained control by virtue
of an .htaccess directive to Apache to use this program to
render the source file. The name of that source file has to
be recovered somehow, and different systems seem to handle
this parameter in different ways. The first parameter to
explore is the REDIRECT_QUERY_STRING, which, if it is
present in the form request, contains secondary parameters to
the rendering operation. If this parameter is not present,
initialize the variable form to an empty value.
3.4.1 Handle REDIRECT QUERY STRING
<handle redirect query string 3.17> =query_string=os.environ['REDIRECT_QUERY_STRING']
form=cgi.parse_qs(query_string)
if form.has_key('debug') and form['debug'][0]=='true':
sys.stderr.write("%s: %s\n" % (tsstring,repr(form)))
debugFlag=True
print "<h1>%s: INDEX.PY version %s</h1>\n" % (tsstring,version)
print "<p>%s: os.environ=%s</p>\n" % (tsstring,repr(os.environ))
print "<p>%s: form=%s</p>\n" % (tsstring,repr(form))
sys.stderr.write("%s: redirect_query string=%s\n" % (tsstring,query_string))
if form.has_key('xml'):
if form['xml'][0]=='true':
sys.stderr.write("%s: %s\n" % (tsstring,repr(form)))
returnXML=True
if
debugFlag:
print "<p>%s: os.environ=%s</p>\n" % (tsstring,repr(os.environ))
print "<p>%s: form=%s</p>\n" % (tsstring,repr(form))
sys.stderr.write("%s: redirect_query string=%s\n" % \
(tsstring,query_string))
elif form['xml'][0]=='convert':
convertXML=True
There are several possibilities for secondary parameters.
The primary one is the debugFlag parameter, which can
be set to true, indicating that debugging information is to
be printed along with the rendering. This is intended for
administrator access only, but as it is harmless, there is
no authentication required.
The other parameter that can be offered at this point is
the xml parameter, with values of true or
convert. The first of these forces no conversion
of the XML, but simply copies it to the browser,
substituting escape sequences for any special XML character
sequences so that it appears as verbatim XML.
The second choice, convert, allows the use of the
rendering engine as an XML-to-HTML converter, in which case
a copy of the converted HTML is saved to a temporary file.
This file can be used subsequently as a statically converted
file as necessary.
3.5 Get Filename from Redirect
<get filename from redirect 3.18> =# get the file name from the redirect environment
if system==MacOSX:
scriptURL='REQUEST_URI'
elif system==Solaris:
scriptURL='REDIRECT_URL'
elif system==Linux:
scriptURL='REDIRECT_URL'
elif system==Ubuntu:
scriptURL='REDIRECT_URL'
if os.environ.has_key(scriptURL):
requestedFile=os.environ[scriptURL]
argpos=
requestedFile.find('?')
if argpos>=0:
requestedFile=
requestedFile[0:argpos]
if debugFlag:
sys.stderr.write("%s: [client %s] requesting %s\n" % \
(tsstring,remoteAdr,
requestedFile))
orgfile=
requestedFile
# analyse file request. If a bare directory, add 'index.xml'
if os.environ.has_key('REDIRECT_STATUS') and \
os.environ['REDIRECT_STATUS']=='404':
res=
htmlpat.match(
requestedFile)
if res:
filename=res.group(1)+'.xml'
requestedFile=filename
dir=relcwd=""
res=filepat.match(
requestedFile)
if res:
dir=res.group(1)
relcwd=dir
# protocol for relcwd:
# no subdir => relcwd = '' (empty)
# exists subdir => relcwd = subdir (no leading or trailing slash)
if dir!="":
requestedFile=dir+'/'+res.group(2)
else:
requestedFile=res.group(2)
filename=res.group(2)
else:
# not ajh (sub)directory, extract full directory path
dir=os.path.dirname(
requestedFile)
relcwd=dir
filename=os.path.basename(
requestedFile)
if
debugFlag:
print "<p>%s: dir,requestedFile,relcwd to process = %s,%s,%s</p>" % \
(tsstring,dir,
requestedFile,relcwd)
3.6 Check for Abbreviated URL
<check for abbreviated URL 3.19> =
3.7 Make File and Dir Absolute
<make file and dir absolute 3.20> =
3.8 Check for HTML Request
We now have a requestedFile name for the document to
be rendered. We need to investigate this file to see how it is
to be rendered. In particular, it may be an HTML file
(indicated by a .html extension), or it may be an XML
file previously rendered and cached. In these cases, we do
not need to do any XML conversion, and the flag
alreadyHTML is set true if it is an HTML file, or the
flag cachedHTML is set true if it is a cached converted
XML to HTML file.
<check for HTML request 3.21> =res=
htmlpat.match(
requestedFile)
if res:
# we have an HTML request, check if it exists
if os.path.exists(
requestedFile):
# exists, use that
alreadyHTML=True
if debugFlag: print "requested file %s is already html<br/>" % (htmlpath)
else:
# doesn't exist, convert from HTML
filename=res.group(1)+'.xml'
requestedFile=filename
This code now also checks for a cached version of the XML
file, as per the following fragment.
<check for HTML request 3.22> =if not
alreadyHTML:
patn="(%s/)(.*).xml" % (BASE)
if debugFlag:
print "<p>matching xml=%s with pattern=%s<br/>" % (
requestedFile,patn)
res=re.match(patn,
requestedFile)
if res:
base=res.group(1); path=res.group(2)
if debugFlag: print "matched BASE=%s,path=%s<br/>" % (base,path)
htmlpath="%s%s.html" % (
HTMLS,path)
if os.path.exists(htmlpath):
htmlstat=os.stat(htmlpath)
xmlstat=os.stat(
requestedFile)
htmlmod=htmlstat.st_mtime
xmlmod=xmlstat.st_mtime
if xmlmod < htmlmod and not form:
# cached version is newer use that
if debugFlag: print "using cached file %s<br/>" % (htmlpath)
requestedFile=htmlpath
cachedHTML=True
else:
if debugFlag: print "no cached version of %s<br/>" % (
requestedFile)
else:
if debugFlag: print "requested file %s is not XML<br/>" % (
requestedFile)
Unless the file being retrieved is already an HTML file,
check to see if we have a cached HTML version of this (XML)
file. Note that any parameters to th http request (indicated
by a non-empty form value) will abort the caching
process, and force a reload of the XML file.
3.9 Determine the Host and Server Environments
<determine the host and server environments 3.23> =# determine which host/server environment
host=commands.getoutput('hostname')
host=re.split('\.',host)[0] # break off leading part before the '.' char
# discard bots ignoring robots.txt
try:
clientIP=os.environ["REMOTE_ADDR"]
if clientIP in ['66.249.77.77','217.79.181.38']:
print "Content-type: text/html\n"
print "<H1>NOT AVAILABLE</H1>"
sys.exit(1)
except KeyError:
pass
try:
host=os.environ["HOSTNAME"]
except KeyError:
cmd='/bin/hostname'
pid=Popen(cmd,shell=True,stdout=PIPE,stderr=PIPE,close_fds=True)
host=pid.communicate()[0].strip()
try:
server=os.environ["SERVER_NAME"]
except KeyError:
server='localhost'
try:
docRoot=os.environ["DOCUMENT_ROOT"]
except KeyError:
docRoot='/Users/ajh/www'
if os.environ.has_key("SCRIPT_URL"):
URL=os.environ["SCRIPT_URL"]
elif os.environ.has_key("REDIRECT_URL"):
URL=os.environ["REDIRECT_URL"]
else:
URL="We got a problem"
# determine the server and host names
#
# the server is the address to which this request was directed, and is
# useful in making decisions about what to render to the client.
# Examples are "localhost", "www.ajh.id.au", "chairsabs.org.au".
#
# the host is the machine upon which the server is running, and may be
# different from the server. This name is used to determine where to
# store local data, such as logging information. For example, the
# server may be "localhost", but this can run on a variety of hosts:
# "murtoa", "dimboola", dyn-13-194-xx-xx", etc.. Incidentally, hosts
# of the form "dyn-130-194-xx-xx" are mashed down to the generic "dyn".
MacOSX='MacOSX' ; Solaris='Solaris' ; Linux="Linux" ; Ubuntu='Ubuntu'
ostype=system=MacOSX # unless told otherwise
if server in ["localhost"]:
pass
elif server in ['ajh.co','www.ajh.co','spencer','ajh.id.au','www.ajh.id.au']:
host='spencer'
ostype=Linux
system=Ubuntu
elif server in ['ajhurst.org','www.ajhurst.org',\
'albens.ajhurst.org','45.55.18.15',\
'njhurst.com','www.njhurst.com']:
server='ajhurst.org'
host='albens'
ostype=Linux
system=Ubuntu
elif server in ['wolseley','wolseley.home.gateway']:
server='wolseley'
host='wolseley'
ostype=Linux
system=Ubuntu
elif server in ['dimboola','dimboola.local','dimboola.ajh.id.au']:
host='dimboola'
elif server in ['echuca','echuca.local','echuca.ajh.id.au']:
host='echuca'
elif server in ["www.csse.monash.edu.au"]:
host='csse'
ostype='Solaris'
system=Solaris
elif server in ['eregnans.ajhurst.org','regnans.njhurst.com']:
host='eregnans'
ostype=Linux
system=Ubuntu
elif server in ['cahurst.org']:
host='albens'
ostype=Linux
system=Ubuntu
elif server in ['glenwaverleychurches.org','www.glenwaverleychurches.org']:
host='albens'
ostype=Linux
system=Ubuntu
elif server in ['cerg.infotech.monash.edu.au','cerg.csse.monash.edu.au']:
host='cerg'
ostype=Linux
server='cerg.infotech.monash.edu.au' # repoint the server address
system=Ubuntu
elif server in ['10.0.0.110']:
server='www.ajh.id.au'
host='dimboola'
elif server[0:11]=='dyn-130-194-':
server='localhost'
host='dyn'
else:
sys.stderr.write("server/host values not recognized\n")
sys.stderr.write("(supplied values are %s/%s)\n" % (server,host))
ostype=Linux
system=Ubuntu
sys.stderr.write("(assuming (ostype,system)=(%s,%s)\n" % (ostype,system))
3.10 Get Default XSLT File
<get default XSLT file 3.24> =# collect the XSLT file name from the .htaccess referent
if os.environ.has_key('QUERY_STRING'):
query_string=os.environ['QUERY_STRING']
else:
query_string='xslfile=%s/lib/xsl/ajhwebdoc.xsl&/~ajh/index.xml' % (BASE)
if
debugFlag:
sys.stderr.write("%s: query string=%s\n" % (tsstring,query_string))
form2=cgi.parse_qs(query_string)
if
debugFlag:
print("<p>%s: form2=%s</p>\n" % (tsstring,repr(form)))
if form2.has_key('xslfile'):
xslfile=form2['xslfile'][0]
if
debugFlag:
print "<p>%s: got xslfile=%s</p>\n" % (tsstring,xslfile)
3.11 Scan for Locally Defined XSLT File
<scan for locally defined XSLT file 3.25> =# Check the requested file for a local stylesheet. We also scan the
# entire file, replacing any symbolic references to $WEBDIR with the
# full path for the current machine. Note that the DOCTYPE statement
# must start a line by itself.
try:
filed=open(
requestedFile)
text='' ; linecount = 0
trackXML=
debugFlag and not (
alreadyHTML or
cachedHTML)
while 1: # keep scanning file until we find no more XML directives
line=filed.readline()
if line=='': # this is EOF, so quit
if linecount==0:
print "(empty file)"
break
linecount+=1
line=line.strip() # remove NL
text+=' '+line
if trackXML:
print "<p>read line='%s'" % (cgi.escape(line))
# check if end of directives, indicated by normal element tag start
res=re.match('<[^?!]',line)
if res:
break
if trackXML:
print "<p>text read='%s'" % (cgi.escape(text))
res=re.match('.*(<\?xml-stylesheet)(.*?)(\?>)',text)
if res:
parms=res.group(2)
# now we have the stylesheet parameters
res=re.match('.*href="(.*?)"',parms)
if res:
# extract filename
xslfile=res.group(1)
xslfile=re.sub('(\$WEBDIR)',WEBDIR,xslfile)
if
debugFlag:
print "<p>%s: stylesheet in xml file, href=%s</p>" % (tsstring,xslfile)
<check if xslfile more recent than cached version 3.26>
else:
if trackXML:
print "<p>Did not find stylesheet href in %s" % (parms)
else:
if trackXML:
print "<p>Did not find stylesheet reference in %s" % (cgi.escape(text))
filed.close()
except:
print """
<h1>Sorry!! (Error 404)</h1>
<p>While processing your request for file %s,<br/>
it was found that the corresponding XML file %s does not exist</p>
<p>Please check that the URL is correct</p>
Exception values: (%s,%s,%s)
""" % (orgfile,
requestedFile,sys.exc_type,sys.exc_value,sys.exc_traceback)
sys.exit(0)
#newfiled.close()
<check if xslfile more recent than cached version 3.26> =localXSLfile=re.sub('file://','',xslfile)
try:
xslmod=os.stat(localXSLfile)
if htmlmod < xslmod:
cachedHTML=False
if debugFlag: print "<p>XSL newer than HTML, reloading</p>"
except: # ignore any errors from this
pass
Look at modification time of XSL file. If it is more recent
than the cached HTML file, we must re-convert the XML file.
3.12 Determine XSLT File
<determine xslt file 3.27> =# have we got an xslfile yet?
htacc=None
if xslfile=="":
# no, so check all .htaccess
# first grab directory
while len(dir)>=len(BASE):
if
debugFlag:
print "<p>directory=%s</p>\n" % (dir)
if os.path.isfile(dir+"/.htaccess"):
htacc=open(dir+"/.htaccess")
if
debugFlag:
print "<p>found .htaccess in directory %s</p>" % (dir)
break
else:
dir=os.path.dirname(dir)
if htacc:
for line in htacc.readlines():
res=xslspec.match(line)
if res:
xslfile=res.group(1)
if xslfile[0] != '/':
xslfile=BASE+'/'+xslfile
break
if
debugFlag:
print "<p>found xslfile %s in .htaccess</p>" % (xslfile)
if system==Solaris:
xslfile=re.sub('/home/ajh'+'/www','/u/web/homes/ajh',xslfile)
if xslfile[0]!='/' and not (xslfile[0:5]=='file:'):
xslfile='/u/web/homes/ajh/'+xslfile
3.13 Update Counter
Compute the name of an XML counter file which contains a
counter element with subelements value and date.
The value element contains the current count value, and the
date element is the date on which this XML file was
initialised. We read the current count from that file,
increment it, and update the file. This file is used by most
xslt translations to output an access count in the footer. It
is also used by the site map program to compute the intensity
of accesses to this web page.
It was fortuitous, but this counter also keeps track of HTML
accesses, both where an HTML file is the initial request, and
where it is a cached version of the corresponding XML file.
Since the XML files have their own counters included by the
XSLT translator, the count attached to the HTML rendering
allows a comparision of how many accesses are to the cached
copy (the difference between the two).
For example, suppose the XML rendering gives 986 references,
and the HTML rendering cites 993 references. The the cached
HTML page has itself been referenced 7 times since it was
first cached.
<update counter 3.28> =counterName=re.sub("/~ajh/",'',relfile)
counterName=re.sub("^/",'',counterName)
extnPattern=re.compile("(.xml)|(.html)")
counterName=re.sub(extnPattern,'',counterName)
counterName=COUNTERS+re.sub("/","-",counterName)
First we process relfile to find the counter name.
Remove any extension, and replace all slash path separators
with minus signs.
(Strictly speaking, the first sub is not required, but
I've left it in, as it does no harm.)
<update counter 3.29> =newCounterStr='<?xml version="1.0"?>\n'
newCounterStr+='<counter><value>0</value><date>%s</date></counter>' % todayStr
try:
counterFile=open(counterName,'r')
dom=xml.dom.minidom.parse(counterFile)
counterFile.close()
except IOError:
dom=xml.dom.minidom.parseString(newCounterStr)
except xml.parsers.expat.ExpatError:
dom=xml.dom.minidom.parseString(newCounterStr)
except:
print "Unexpected error:", sys.exc_info()[0]
raise
Now try to read the counter XML file. The file may not exist
if this is the first time we have accessed this page since
this mechanism was set up, so we must capture that error, and
any error arising from attempting to parse the XML, and create
a new counter file, with value initialised to zero, and
date initialised to today's date.
<update counter 3.30> =# now extract count field and update it
countNode=dom.getElementsByTagName('value')[0]
if countNode.nodeType == xml.dom.Node.ELEMENT_NODE:
textNode=countNode.firstChild
if textNode.nodeType == xml.dom.Node.TEXT_NODE:
text=textNode.nodeValue.strip()
countVal=int(text)
countVal=countVal+1
textNode.nodeValue="%d" % (countVal)
countDate='(unknown)'
countNode=dom.getElementsByTagName('date')[0]
if countNode.nodeType == xml.dom.Node.ELEMENT_NODE:
textNode=countNode.firstChild
if textNode.nodeType == xml.dom.Node.TEXT_NODE:
countDate=textNode.nodeValue.strip()
<update counter 3.31> =# write updated counter document
try:
counterFile=open(counterName,'w')
except IOError:
print "could not open %s" % (counterName)
counterName='/home/ajh/local/localhost/counters/index'
counterFile=open(counterName,'w')
domString=dom.toxml()
counterFile.write(domString)
counterFile.close()
3.14 Process File
<process file 3.32> =# define the parameters to the translation
filestat=os.stat(
requestedFile)
filemod=filestat.st_mtime
dtfilemod=datetime.datetime.fromtimestamp(filemod)
dtstring=dtfilemod.strftime("%Y%m%d:%H%M")
parms=""
parms+="--param xmltime \"'%s'\" " % (dtstring)
parms+="--param htmltime \"'%s'\" " % (tsstring)
parms+="--param filename \"'%s'\" " % (filename)
parms+="--param relcwd \"'%s'\" " % (relcwd)
parms+="--param URL \"'%s'\" " % (URL)
parms+="--param today \"'%s'\" " % (todayStr)
parms+="--param host \"'%s'\" " % (host)
parms+="--param server \"'%s'\" " % (server)
parms+="--param base \"'%s'\" " % (BASE)
for key in form:
value=form[key][0]
parms+="--param "+key+" \"'%s'\" " % (value)
if
debugFlag:
sys.stderr.write("%s: xml file modified at %s\n" % (tsstring,dtstring))
<process file 3.33> =
Decide what to with the file. There are 3 choices:
- return the raw XML. This means escaping all the active
characters, and printing the file verbatim.
- The file is HTML, either because of an explicit HTML
request, or a cached HTML file previously translated has
been found. Again, the file is rendered verbatim, this
time without escaping the active characters.
- It is an XML file, and it needs translation. Call the
XSLT processor to do that (chunk
<process an XML file 3.35,3.36,3.37>).
<render the HTML file 3.34> =rawHTMLf=open(
requestedFile,'r')
for line in rawHTMLf.readlines():
print line,
print '<P><SPAN STYLE="font-size:80%%">'
print '%d accesses since %s, ' % (countVal,countDate)
print 'HTML cache rendered at %s</SPAN>' % (dtstring)
if
cachedHTML:
os.utime(
requestedFile,None) # touch the file
Note that each line from the HTML file is printed without
additional line breaks.
3.14.1 Process an XML File
<process an XML file 3.35> =# start a pipe to process the XSLT translation
cmd=XSLTPROC+" --xinclude %s%s %s " % (parms,xslfile,
requestedFile)
#(pipein,pipeout,pipeerr)=os.popen3(cmd)
pid=Popen(cmd,shell=True,stdout=PIPE,stderr=PIPE,close_fds=True)
(pipeout,pipeerr)=(pid.stdout,pid.stderr)
if
debugFlag:
cwd=os.getcwd()
print "<p>%s: (cwd:%s) %s</p>" % (tsstring,cwd,cmd)
sys.stderr.write("(cwd:%s) %s: %s\n" % (cwd,tsstring,cmd))
# report the fact, and the context (debugging purposes)
if
debugFlag:
print "%s: converting %s with %s\n" % (tsstring,
requestedFile,xslfile)
Run the pipe to perform the translation. Note that this
step requires an inordinate amount of time on some servers
(sequoia in particular), and was the prompt for
including the caching mechanism.
<process an XML file 3.36> =# process the converted HTML
convertfn="/home/ajh/www/tmp/convert.html"
if
convertXML:
try:
htmlfile=open(convertfn,'w')
except:
msg="couldn't open HTML conversion file %s" % convertfn
sys.stderr.write("%s: %s\n" % (tsstring,msg))
convertXML=False
<process an XML file 3.37> =# check that directory exists
dirpath=os.path.dirname(htmlpath)
if not os.path.isdir(dirpath):
os.makedirs(dirpath,0777)
htmlfile2=open(htmlpath,'w')
for line in pid.stdout.readlines():
print line,
htmlfile2.write(line)
if
convertXML:
htmlfile.write("%s\n" % line)
if
convertXML:
htmlfile.close()
htmlfile2.close()
os.chmod(htmlpath,0666)
<deal with any conversion errors 3.38>
pipeout.close(); pipeerr.close()
Note that in copying the rendered HTML version, we retain
the lines as is, and make sure that they are rendered
without any additional (or deleted) new lines.
3.14.1.1 Deal with any Conversion Errors
<deal with any conversion errors 3.38> =errs=[]
for line in pipeerr.readlines():
errs.append(line)
logfile=PRIVATE+'/xmlerror.log'
logfiled=open(logfile,'a')
if errs:
logfiled.write("%s: ERROR IN REQUEST: %s\n" % (tsstring,
requestedFile))
print "<HR/>\n"
print "<H3>%s: MESSAGES GENERATED BY: %s</H3>\n" % (tsstring,
requestedFile)
print "<PRE>"
for errline in errs:
#logfiled.write("%s: %s" % (tsstring,errline))
errline=cgi.escape(errline)
errline=errline.rstrip()
print "%s: %s" % (tsstring,errline)
print "</PRE>"
print "<p>Please forward these details to "
print "<a href='mailto:ajh@csse.monash.edu.au'>John Hurst</a>"
else:
logfiled.write("%s: NO ERRORS IN %s\n" % (tsstring,
requestedFile))
logfiled.close()
4. The Train Image Viewer viewtrains.py
The literate code for this program has been moved to
ViewTrains
5. The Rankings Display Program ranktrains.py
The literate code for this program has been moved to
ViewTrains
6. The Train Ranking Module rank.py
The literate code for this program has been moved to
ViewTrains
7. The Web Server Module webServer.py
"webServer.py" 7.1 =#!/usr/bin/python
# online.py
#
# This program looks at all machines known to the ajh network, and
# examines them to see if they are on-line.
#
# version 1.0.0 20160622:104949
#
# put imports here: those named are required by this template
import datetime
import subprocess
import socket
import re
import getopt
import os,os.path
import sys
import shutil
# define globals here
debug=0; verbose=0
machines=['spencer',
#'spencer.ajh.co',
'dimboola',
#'ajh.id.au',
#'wolseley',
'hamilton.local',
#'bittern',
#'echuca',
'lilydale',
#'albens.ajhurst.org'
]
macosx=['dimboola','ajh.id.au','hamilton.local','MU00087507X','bittern','echuca']
linux=['spencer','spencer.ajh.co','wolseley','lilydale','lilydale.local','albens.ajhurst.org']
thismachine=socket.gethostname()
ignoreOut=open('/dev/null','w')
# define usage here
def usage():
print """
This module is intended for use in web page delivery programs.
It does not serve any useful purpose in its own right.
<flags>= [-d|--debug] print debugging information
[-v|--verbose] print more debugging information
[-V|--version] print version information
The debug flags give (verbose) output about what is happening.
"""
# define global procedures here
# (there are none)
# define the key class of this module
class location():
def __init__(self,server=thismachine):
# determine the server and host names
#
# the server is the address to which this request was directed, and is
# useful in making decisions about what to render to the client.
# Examples are "localhost", "www.ajh.id.au", "chairsabs.org.au".
#
# the host is the machine upon which the server is running, and may be
# different from the server. This name is used to determine where to
# store local data, such as logging information. For example, the
# server may be "localhost", but this can run on a variety of hosts:
# "murtoa", "dimboola", dyn-13-194-xx-xx", etc.. Incidentally, hosts
# of the form "dyn-130-194-xx-xx" are mashed down to the generic "dyn".
MacOSX='MacOSX' ; Solaris='Solaris' ; Linux="Linux" ; Ubuntu='Ubuntu'
ostype=system=MacOSX # unless told otherwise
host=server
if server in ["localhost"]:
pass
elif server in ['ajh.co','www.ajh.co','spencer']:
host='spencer'
ostype=Linux
system=Ubuntu
elif server in ['albens','albens.ajhurst.org','45.55.18.15',\
'ajhurst.org','www.ajhurst.org',\
'njhurst.com','www.njhurst.com']:
#server='ajhurst.org'
host='albens'
ostype=Linux
system=Ubuntu
elif server in ['dimboola','dimboola.local',\
'ajh.id.au','dimboola.ajh.id.au']:
host='dimboola'
elif server in ['wolseley','wolseley.home.gateway']:
server='wolseley'
host='wolseley'
ostype=Linux
system=Ubuntu
elif server in ['echuca','echuca.local','echuca.ajh.id.au']:
host='echuca'
elif server in ["www.csse.monash.edu.au"]:
host='csse'
ostype='Solaris'
system=Solaris
elif server in ['eregnans.ajhurst.org','regnans.njhurst.com']:
host='eregnans'
ostype=Linux
system=Ubuntu
elif server in ['cahurst.org']:
host='albens'
ostype=Linux
system=Ubuntu
elif server in ['glenwaverleychurches.org','www.glenwaverleychurches.org']:
host='albens'
ostype=Linux
system=Ubuntu
elif server in ['cerg.infotech.monash.edu.au','cerg.csse.monash.edu.au']:
host='cerg'
ostype=Linux
server='cerg.infotech.monash.edu.au' # repoint the server address
system=Ubuntu
else:
sys.stderr.write("server/host values not recognized\n")
sys.stderr.write("(supplied values are %s/%s)\n" % (server,host))
ostype=Linux
system=Ubuntu
sys.stderr.write("(assuming (ostype,system)=(%s,%s)\n" % (ostype,system))
self.ostype=ostype
self.system=system
self.server=server
self.host=host
pass
# define the main program here
def main(s):
loc=location(s)
print "I have gleaned that:"
print " this machine = %s" % (thismachine)
print " this server = %s" % (loc.server)
print " this host = %s" % (loc.host)
print " this ostype = %s" % (loc.ostype)
print " this system = %s" % (loc.system)
pass
if __name__ == '__main__':
(vals,path)=getopt.getopt(sys.argv[1:],'dvV',
['debug','verbose','version'])
for (opt,val) in vals:
if opt=='-d' or opt=='--debug':
debug=1
if opt=='-v' or opt=='--verbose':
verbose=1
if opt=='-V' or opt=='--version':
print version
sys.exit(0)
server=thismachine
if len(path)>0:
server=path[0]
main(server)
The various web server programs all make use of a pair of
values, known as host and server. The host
value defines the machine on which this program is
running, while the server value defines the name by which
the program was invoked (effectively the domain name of the
server). This is necessary, as each program may be invoked by
different domain name paths, and the service rendered may be
different for each path.
While different domain names may invoke services on the same
machine or host, it is never the case that different
machines are invoked to handle a single domain name service.
When invoked as a stand-alone program, a test server
parameter may be passed in to see what the program determines.
This is for testing purposes only, and serves no useful purpose
otherwise.
8. File Caching
8.1 The File Cache Module
"filecache.py" 8.1 ="""A module that writes a webpage to a file so it can be restored at a later time
Interface:
filecache.write(...)
filecache.read(...)
"""
import time
import os
import md5
import urllib
def key(url):
k = md5.new()
k.update(url)
return k.hexdigest()
def filename(basedir, url):
return "%s/%s.txt"%(basedir, key(url))
def write(url, basedir, content):
""" Write content to cache file in basedir for url"""
cachefilen=filename(basedir, url)
fh = file(cachefilen, mode="w")
fh.write(content)
fh.close()
return cachefilen
def read(url, basedir, timeout):
"""Read cached content for url in basedir if it is fresher
than timeout (in seconds)"""
cache=0
fname = filename(basedir, url)
content = ""
if os.path.exists(fname) and \
(os.stat(fname).st_mtime > time.time() - timeout):
fh = open(fname, "r")
content = fh.read()
fh.close()
cache=1
return (content,cache)
This code was adapted from an example given on a web page.
Sorry, I have forgotten the reference.
8.2 Clearing the Cache
This program clears the HTML caches created by the previous
module. It is called independently, and can clear either the
entire cache, or subdirectories of it.
The cache is maintained on a per-machine basis, and the
machine being used is identified by a hostname
call.
"clearWebCache.py" 8.2 =
9. The Makefile
The Makefile handles the nitty-gritty of copying
files to the right places, and setting permissions, etc.
<install python file 9.1> =install.machine: /tmp/index-machine.py
chmod a+x /tmp/index-machine.py
if [ ${HOST} = machine ] ; then \
cp -p /tmp/index-machine.py homedir/www/cgi-bin/index.py; \
else \
rsync -auv /tmp/index-machine.py address:homedir/www/cgi-bin/index.py; \
fi
/tmp/index-machine.py: index.py
sed -e 's#/sw/bin/python#interpreter#' <index.py >/tmp/index-machine.py
"Makefile" 9.2 =RELCWD = /cgi-bin/
WEBPAGE = /home/ajh/www/research/literate
FILES = $(EMPTY)
XSLLIB = /home/ajh/lib/xsl
XSLFILES = $(XSLLIB)/lit2html.xsl $(XSLLIB)/tables2html.xsl
INSTALLFILES = index.py
CGIS = $(INSTALLFILES)
XMLS = $(EMPTY)
DIRS = $(EMPTY)
include $(HOME)/etc/MakeXLP
include $(HOME)/etc/MakeWeb
index.py: web.tangle
chmod 755 index.py
touch index.py
web.tangle web.xml: web.xlp
xsltproc --xinclude -o web.xml $(XSLLIB)/litprog.xsl web.xlp
touch web.tangle
web.html: web.xml $(XSLFILES)
xsltproc --xinclude $(XSLLIB)/lit2html.xsl web.xml >web.html
install: install.${HOST}
web: $(WEBPAGE)/web.html
$(WEBPAGE)/web.html: web.html
cp -p web.html $(WEBPAGE)/web.html
Makefile: web.tangle
install.murtoa: index.py
sed -e 's#/sw/bin/python#/sw/bin/python2.6#' <index.py >/Users/ajh/www/cgi-bin/index.py
<install python file 9.1>(machine='dimboola', address='dimboola', interpreter='/sw/bin/python2.6', homedir='/Users/ajh')
<install python file 9.1>(machine='eregnans', address='eregnans', interpreter='/usr/bin/python', homedir='/home/ajh')
<install python file 9.1>(machine='wolseley', address='wolseley', interpreter='/usr/bin/python', homedir='/home/ajh')
<install python file 9.1>(machine='spencer', address='spencer', interpreter='/usr/bin/python', homedir='/home/ajh')
<install python file 9.1>(machine='csse', address='nexus', interpreter='/usr/monash/bin/python', homedir='/home/ajh')
<install python file 9.1>(machine='albens', address='albens.ajhurst.org', interpreter='/usr/bin/python', homedir='/home/ajh')
The install.system targets are designed to cater
for the variations in interpreters required for each of the
servers installed by the Makefile.
Note that this has not been updated for setting the filecache module.
10. TODOs
- 20081111:103026
- footer links filename is incorrect when a .html URL is
used. Need to check calculation of filename.
11. Indices
11.1 Files
11.2 Chunks
Chunk Name |
Defined in |
Used in |
check for HTML request |
3.21, 3.22
|
3.8 |
check for abbreviated URL |
3.19 |
3.8 |
check for and handle jpegs |
3.13 |
3.6 |
check if xslfile more recent than cached version |
3.26 |
3.25 |
collect HTTP request |
3.16 |
3.7 |
current date |
.2 |
|
current version |
.1 |
3.1 |
deal with any conversion errors |
3.38 |
3.37 |
define global variables and constants |
3.10, 3.11, 3.12
|
3.4 |
define various string patterns |
3.9 |
3.6 |
determine the host and server environments |
3.23 |
3.4 |
determine xslt file |
3.27 |
3.8 |
get default XSLT file |
3.24 |
3.8 |
get filename from redirect |
3.18 |
3.7 |
get jpg file from remote server |
3.14, 3.15
|
3.13 |
handle redirect query string |
3.17 |
3.16 |
install python file |
9.1 |
9.2, 9.2, 9.2, 9.2, 9.2, 9.2
|
make file and dir absolute |
3.20 |
3.8 |
process an XML file |
3.35, 3.36, 3.37
|
3.33 |
process file |
3.32, 3.33
|
3.8 |
render the HTML file |
3.34 |
3.33 |
scan for locally defined XSLT file |
3.25 |
3.8 |
update counter |
3.28, 3.29, 3.30, 3.31
|
3.8 |
11.3 Identifiers
Identifier |
Defined in |
Used in |
HTMLS |
3.11 |
3.22 |
alreadyHTML |
3.10 |
3.21, 3.22, 3.25, 3.33
|
cachedHTML |
3.10 |
3.22, 3.25, 3.33, 3.34
|
convertXML |
3.10 |
3.17, 3.36, 3.36, 3.37, 3.37
|
debugFlag |
3.10 |
3.8, 3.16, 3.17, 3.17, 3.18, 3.24, 3.24, 3.24, 3.25, 3.25, 3.27, 3.27, 3.27, 3.32, 3.35, 3.35
|
htmlpat |
3.9 |
3.18, 3.21
|
requestedFile |
3.16 |
3.8, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.18, 3.19, 3.19, 3.19, 3.20, 3.20, 3.20, 3.21, 3.21, 3.21, 3.22, 3.22, 3.22, 3.22, 3.22, 3.22, 3.25, 3.25, 3.32, 3.33, 3.34, 3.34, 3.35, 3.35, 3.38, 3.38, 3.38
|
returnXML |
3.10 |
3.17, 3.33
|
Document History
20080816:144135 |
ajh |
1.0.0 |
first version under literate programming |
20080817:131040 |
ajh |
1.0.1 |
general restructuring |
20080822:162138 |
ajh |
1.0.2 |
more restructuring |
20081102:164507 |
ajh |
1.1.0 |
added jpg handling and caching |
20081106:134033 |
ajh |
1.1.1 |
added exception handling |
20090507:160328 |
ajh |
1.2.0 |
bug relating to non-~ajh directories fixed;
caching of files implemented.
|
20090701:175934 |
ajh |
1.3.0 |
added code to cache the converted HTML file. Still to
do: creation of subdirectories for cached files. |
20090702:182341 |
ajh |
1.3.1 |
Subdirectories now created. Cached file is touched on
each access |
20090703:105343 |
ajh |
1.3.2 |
some literate tidy ups, and renamed variable
file to requestedFile to disambiguate it.
|
20091203:093814 |
ajh |
1.3.3 |
updated Makefile to install python interpreter
dependent files |
20120530:134427 |
ajh |
1.3.4 |
fix bug in glenwaverleychurches handling of counters |
|
ajh |
1.3.5 |
source code only |
20160221:165415 |
ajh |
1.3.6 |
modified for albens |
20160407:111509 |
ajh |
1.3.7 |
further albens changes for cahurst domain |
<current version .1> = 1.3.6
<current date .2> = 20160221:165516