HouseMade - The Hurst HouseHold Heater Helpmate

A.J.Hurst

Version 1.3.6

20160204:171508

Table of Contents

1 Introduction
1.1 Overview
1.2 TODOs
1.3 History
1.4 Philosophies
2 Key Data Structures
2.1 Edit Warning
2.2 House Definitions Module
3 The Web Interface
3.1 The Flask Application
3.1.1 Define the Various Flask URLs and Calls
3.1.2 Define the Showpage Routine
3.1.3 Start the Flask application
3.1.4 Watchdog program for Flask
3.2 The HouseMade module
3.2.1 define the Generate Weather Data routine
3.2.2 define the Generate Solar Data routine
3.2.3 define the Generate Tank Data routine
3.2.4 Collect Date and Time Data
3.2.5 Check the Client Connection
3.2.6 Generate the Web Page Content
3.2.7 Make the Temperature Panel (not currently used)
3.3 The HeatingModule module
3.3.1 Define the heatingData Class
3.3.2 Collect Parameters and Update
3.3.3 Build Widths for Web Page Table
4 The Weather System
4.1 The C Weather Monitor Program
4.2 The Python Interface to the Weather System
4.3 The Weather Logging Process
5 The Heating System
5.1 AdjustHeat
5.2 checkTime.py
5.3 TempLog
6 The Tank System
6.1 Water Tank Logging
6.2 Start the Tank Logging
6.3 Tank Module Functions
7 The Solar System
7.1 logsolar.py
7.2 solar.py
8 The Chook Door
8.1 ChookDoor: load
8.2 ChookDoor: compute
8.3 ChookDoor: save
8.4 ChookDoor: compute ephemeral values
8.5 checkChooks: update the current chook door state
8.6 Chook Door Proving Subsystem
8.7 Chook Door Proving Server
9 The House Computer
9.1 The Current State Interface
9.2 The HouseData Server (obsolete)
9.2.1 HouseData define getTemps
9.2.2 HouseData define maxminTemp
9.3 The startHouseData.sh script
10 The House Data Server and Relay Control System
10.1 The Relay Controller Code
10.2 Relay Server
10.2.1 HouseData define getTank
10.2.2 relayserver: getTimer
10.2.3 relayserver: getState
10.2.4 relayserver: setState
10.2.5 relayserver: setBit
10.2.6 relayserver: setBitOn
10.2.7 relayserver: setBitOff
10.2.8 relayserver: start
10.2.9 relayserver: countDown
10.3 Start the Relay Controller
10.4 The Old Relay Setup
10.5 The Arduino Interface
10.5.1 The Arduino Start Script
11 Hardware Issues
11.1 USB Issues
11.2 USB Resolution
11.2.1 usbFind: USBclass.load
11.2.2 usbFind: USBclass.compute
11.2.3 usbFind: USBclass.device2port
11.2.4 usbFind: USBclass.save
11.3 Current Patchboard Wiring
11.3.1 Cellar Computer Board
11.3.2 Cellar Door Board
11.3.3 House Corner Board
11.3.4 BBQ Board
11.3.5 Restructure Plan
11.3.6 Power Bus
11.4 Current Cat5 Ethernet Wiring
11.4.1 Cellar Patch Board
11.4.2 Study
11.4.3 Family
12 Test Programs
12.1 Check RPC Operation
13 The Log Files
14 (Re)Starting the Server
14.1 Introduction
14.2 Non-Upstart Method (more reliable!)
14.2.1 Sort out USB allocations
14.2.2 Start the Flask Server
14.2.3 Start the Arduino Driver
14.2.4 Start the RelayServer
14.2.5 Start the Tank Logger
14.2.6 Start the Solar Logger
14.2.7 Start the Weather Station Daemon
14.2.8 Start the Chook Door Server
14.2.9 Start the Chook Door Prover
14.3 The New UpStart Configuration
14.3.1 USB Allocation
14.3.2 Flask Server
14.3.3 Arduino
14.3.4 Relay Server
14.3.5 Tank Logging
14.3.6 Solar Logging
14.3.7 Weather Station
14.3.8 Regular Service Activity
15 The Cron Jobs
16 The Makefile (separate file)
16.1 Makefile: install bastille
17 Document History
18 Indices
18.1 Files
18.2 Chunks
18.3 Identifiers


1. Introduction

These programs have been distilled from Nathan Hurst's original Nautilus Shell suite of programs. The main differences are a) the change of name, b) the documentation, and c) consistency in logging structures (where possible).

1.1 Overview

There are a number of programs in this suite, and they are grouped into the following categories:

The Web Interface (operational)
Provides an easy to use interface to the program suite, using two key programs: house and timer. These both invoke the house computer using the Flask protocol.
The Heating System (operational)
This subsystem controls the house heating system. Both the temperature and timing may be controlled: the temperature in steps of 1 degree Celsius, form 10 to 26 degrees, and the time in steps of 5 minutes from midnight to midnight, 7 (distinct) days of the week. Up to 7 time blocks per day are permitted.
The Tank System (operational)
Provides monitoring of the water storage facilities.
The Solar System (operational)
Provides monitoring of the solar photovoltaic systems, together with the inverter and UPS sub-systems.
The Chook Door System (operational)
Controls the opening and shutting of the Chook House Door.
The House Computer (obsolete)
Provides logging of house data, as well as an RPC interface to access the logged data. This functionality has been subsumed by being incorporated into each subsystem, and a separate logging system is no longer used.
The House Data Server and Relay Control System (operational)
Uses an Arduino driving 8 relays to switch the various circuits.

1.2 TODOs

1.3 History

The original House Computer was set up on an Intel 386 box, named redfern, in keeping with my philosophy of naming all my computers after railway junctions. But redfern did not have enough expansion capability, and used the rapidly dating EISA bus architecture, so it was not long before it was replaced with a larger chassis using a 486 and PCI bus. This machine was named central, and was the mainstay of the house computer system for many years.

The earliest evidence for the operation of these two systems is a file dated 23 May 1999, which I believe was written for the central system. Whatever, by mid 2001 the central system was certainly running, which makes me think that redfern dates are from early 1999, while central was probably commissioned in late 1999 to early 2000. Central was a single system, and ran a suite of programs known as Nautilus (aka "Water Shell", originally because it was a "shell" controlling just the garden watering operations), serving both the web page control systems (through Apache and locally written cgi scripts), as well as the various logging subsystems (temperature, humidity, solar panel insolation, and rainwater tank levels.

Central's disk system was the weakest link in the system, since it died in June 2012, some 12 years after commissioning. This is regarded as a fairly robust operation, and set the benchmark for future systems.

Before it died, however, work had been underway to replace the logging operations through a low-power mini-controller 486 based system, known as garedelyon, with the original intention to move all functionality to garedelyon. However, garedelyon's operating system was not up to running a full web server, and so it became just the data logging component of the system. Garedelyon had several RS232 and USB ports, and took over the responsibility for performing all the logging operations. A remote RPC mechanism allowed the web server system to communicate data between the two systems. Once central had died, the web serving functions were moved to various other machines, ending up on an old PowerBook Apple laptop, known as ringwood.

In January 2015, while we we away overseas, ringwood's battery died, taking the system, including the mini-controller garedelyon and weather station, with it. While I had a spare mini-controller, it was not worth replacing the laptop, and it was decided to move the entire system back to a single controller, based upon the new Beaglebone Black that I had acquired while overseas. This new machine was known as orsay.

In the meantime, while the hardware for orsay was being developed, my Acer laptop known as lilydale was pressed into service, this time running a limited number of functions. Part of this limitation was due to there no longer being any way to communicate with RS232 ports, as used by the weather/tank/solar logging systems on garedelyon. A major part of the hardware redesign occasioned by the switch to using a Beaglebone was the need to run a USB Hub, and to connect all the RS232 ports via RS232/USB dongles.

The re-design of the system was sufficiently complete by mid April 2015 to bring it on-line. Major changes include moving all functionality to the one system (thus reverting to a similar framework to the original Central/Nautilus system), and a shift to using Flask as the main web interface. This major rewrite was renamed HouseMade because of its much wider functionality, and now controls all of the house heating, the watering system, logging and display of house data (solar, water tank), the web interfaces, and most recently, the chook house door. Documentation for this system is this current document.

Unfortunately orsay was a little unreliable, crashing more frequently than it should, and eventually dying altogether in mid Jun 2015 (exact date not recorded). The replacement machine (known as bastille) suffered similar problems, and failed on 10 Jul 2015. The reasons for these failures are not clear, but are thought to be related to power supply instabilities.

Rather than risk another $100 piece of hardware, I have reverted (ostensibly temporarily) to using my Acer laptop, running Ubuntu 14.04, "Trusty Tahr". This did require a few days work to get it going properly, but there have been a number of significant improvements as a consequence:

  1. The tank logging has been migrated to a Python program, thus creating the potential for some more smarts in that subsection.
  2. The USB issues previously identified have been resolved. See section USB Resolution. These issues were all a consequence of the need to eliminate all RS232 code, and switch to a USB only system. The Beaglebone Black is known to have a significant bug in its USB subsystem, and this may have contributed to the unreliable behaviour reported above.
  3. The chook door mechanism has been implemented, and is operational.
  4. The hostname had been hardcoded into the code, since I did not expect that it would be changing so rapidly. It has now been factored out, and replaced by the abstract title of central, a tribute to the original system which lasted for some 12 years.

1.4 Philosophies

The house computer complex is just that - complex. Some design principles are in order. One of the difficulties of design has been the need to maintain several different systems, for different reasons. The main systems are (in abstract terms): the data logger, the house controller, and the house web server. Currently the system is structured so that all of these functions are undertaken by the machine lilydale (see previous section History).

The first principle is then all data logging to be handled by the data logging system (as far as possible). Where this is not possible, the system responsible for collecting the data should transfer it to the logging system as soon as possible, and the logging system should be regarded as the definitive source for any data requests. While this principle was originally framed to permit it to be handled by a separate machine, it can reside anywhere on the house network.

The second principle is that any request to change the state of the house must be passed through the house controller, even if it is not the final direct control mechanism. The state of the house is defined by the state of the (currently) 12 relays controlled by the controller system. Again, there is no requirement that this functionality be co-located with the others.

The third principle is that all web traffic is handled by the web system. A key factor in deciding how this functionality is handled is whether the web server is secure enough (can external users change the heating or open the chook door, for example), and secondly whether it could handle the additional traffic imposed by an externally available web server.

2. Key Data Structures

2.1 Edit Warning

Provide a simple inclusion to flag that all derived files are extracted from this file, rather than stand-alone.

<edit warning 2.1> =
## ********************************************************** ## * do NOT EDIT THIS FILE! * ## * Use $HOME/Computers/House/HouseMade.xlp instead * ## **********************************************************
Chunk referenced in 3.11 4.1 5.1 5.2 5.3 6.1 6.2 7.1 7.2 8.1 8.6 8.7 8.8 9.2 9.5 11.2 11.7 12.1 15.1

2.2 House Definitions Module

The house definitions module, HouseDefinitions.py, gathers together in one place those constants that are common to all systems. Note that no shared variables may be handled by this module, since it is not shared across the various systems as a single instance. (All declared "variables" are actually constants, but python does not allow explicit constant declarations.)

Following a second failure of a Beaglebone, the house computer has reverted to the Acer laptop. This is now possible because of all the work done in getting the USB data inputs working. But it has also forced my to rethink the heavy use of hardcoding the machine name into all these script, hence the new definition of CENTRAL (the name is taken from the original house server).

"HouseDefinitions.py" 2.2 =
import xmlrpclib CENTRAL="lilydale" # system-wide definition of the house-controlling relay complement RelayNames=[ 'ChookUp', # 0 - the order of these is important 'ChookDown', # 1 'Unused2', # 2 'TopUp', # 3 'BottomVegBed', # 4 'MiddleVegBed', # 5 'TopVegBed', # 6 'CarportVegBed', # 7 'RainForest', # 8 'Woo2Plas', # 9 'FloodNDrain', # 10 'Heating' # 11 ] NumberOfRelays = len(RelayNames) RelayTable={} for i in range(NumberOfRelays): RelayTable[RelayNames[i]]=i ThermostatSetting=19 NTempBlocks=7 # max number of distinct temperature blocks allowed HServer='http://%s:5000/heating' % (CENTRAL) MServer='http://%s:5000/house' % (CENTRAL) colours=['#00f', # 10 '#04f','#08f','#0cf','#0ff', # 11-14 '#0fc','#0f8','#0f4','#0f0', # 15-18 '#4f0','#8f0','#cf0','#ff0', # 19-22 '#fc0','#f80','#f40','#f00'] # 23-26 CentralGood=True Central=xmlrpclib.ServerProxy('http://%s:8001' % CENTRAL) # check that the server is running by testing one of its interfaces try: Central.getState() except: # bad response, let users know CentralGood=False def setColourOld(temp): # return colours[temp-10] if temp>=ThermostatSetting: return 'red' else: return 'blue' def setTemperatureOld(arg): t=int(arg) if t>ThermostatSetting: t=ThermostatSetting if t<ThermostatSetting: t=10 return t def setColour(temp): return colours[temp-10] def setTemperature(arg): t=int(arg) return t

The routines setColour and setTemperature are defined to localize these two calculations for the house and timer modules. They will be revised once the temperature adjustment system is rebuilt to its full potential.

3. The Web Interface

The web interface is Flask application running on the house computer lilydale, and providing a convential web page via a port 5000 call (the Flask port number), and interfaces to the house and timer modules through house and heating respectively. The figure '10' in setTemperature is just the lowest temperature that is displayed by the timer web interface.

To preserve security of the system, and prevent unauthorized access, this web server will only operate house functions if it is invoked from a machine on the 10.0.0 network (the private house network). In the longer term, username/password authority may be added.

3.1 The Flask Application

"lilydaleFlask.py" 3.1 =
#!/usr/bin/python from flask import Flask,request import cgi import datetime import os import os.path import re from subprocess import PIPE,Popen import xml.dom.minidom import currentState import HouseMade import HouseDefinitions #import TimerModule import HeatingModule XSLTPROC="/usr/bin/xsltproc" COUNTERS="/home/ajh/local/central/counters" debug=True logging=True def logMsg(msg,NewLine=True): if NewLine: msg+='\n' if logging: logfile=open('/home/ajh/logs/central/flask.log','a') logfile.write(msg) logfile.close() else: print msg, app=Flask(__name__) <Flask: define the various flask URLs and calls 3.2> <Flask: define the showpage routine 3.3> if __name__=='__main__': app.debug=debug app.run(host='10.0.0.112') logfile.close()

3.1.1 Define the Various Flask URLs and Calls

<Flask: define the various flask URLs and calls 3.2> =
@app.route('/house') def house(): argDict=request.args.to_dict(flat=False) remadr=request.environ['REMOTE_ADDR'] server=request.environ['SERVER_NAME'] logMsg("%s@%s: house arguments=%s" % (server,remadr,argDict)) return HouseMade.house(remadr,server,argDict) @app.route('/weather') def weather(): argDict=request.args.to_dict(flat=False) remadr=request.environ['REMOTE_ADDR'] logMsg("%s: weather arguments=%s" % (remadr,argDict)) return HouseMade.weather(argDict) @app.route('/heating') def heating(): argDict=request.args.to_dict(flat=False) remadr=request.environ['REMOTE_ADDR'] logMsg("%s: heating arguments=%s" % (remadr,argDict)) return HeatingModule.heating(logMsg,remadr,argDict) @app.route('/solar') def solar(): argDict=request.args.to_dict(flat=False) remadr=request.environ['REMOTE_ADDR'] logMsg("%s: solar arguments=%s" % (remadr,argDict)) return HouseMade.solar(argDict) @app.route('/tank') def tank(): argDict=request.args.to_dict(flat=False) remadr=request.environ['REMOTE_ADDR'] logMsg("%s: tank arguments=%s" % (remadr,argDict)) return HouseMade.tank(argDict) @app.route('/chooks') def chooks(): render="" house=currentState.HouseState() house.load() state=house.get('chookDoorProve').upper() if state=='OPEN': colour='red' else: colour='green' content='<meta http-equiv="Refresh" content="60">\n' content+="<table border='1' width='100%%' height='100%%' bgcolor='%s'>\n" % (colour) content+="<tr width='100%%'>\n" content+="<td align='center' style='font-size:100px'>\n" content+="%s\n</td>\n</tr>\n</table>" % (state) render="<HTML>\n<BODY>\n%s\n</BODY>\n</HTML>\n" % (content) return render
Chunk referenced in 3.1

These two routines define the principal house web pages: house is the general house page, and timer is the heating timer control page. Note that timer has now (v1.3.0) been replaced by heating.

3.1.2 Define the Showpage Routine

The following routine provides a general purpose webpage server

<Flask: define the showpage routine 3.3> =
@app.route('/<path:pagename>') def showpage(pagename): now=datetime.datetime.now() debug=True tsstring=now.strftime("%Y%m%d:%H%M") todayStr=now.strftime("%d %b %Y") #logMsg("%s: looking for %s" % (tsstring,pagename)) filepat=re.compile('/home/ajh/www/?(.*)/([^/]*)$') filename='index.xml' pagename=re.sub('~ajh/','',pagename) relfile=pagename requestedFile="/home/ajh/www/%s" % pagename if os.path.isdir(requestedFile): requestedFile+="/" if requestedFile[-1]=='/': requestedFile+="index.xml" relfile+="index.xml" #logMsg("%s: Now looking for %s" % (tsstring,requestedFile)) if not os.path.exists(requestedFile): return 'Error 404: I cannot find your requested file %s' % (requestedFile) else: <Flask: check what kind of file is requested 3.4> <Flask: compute the current working directory 3.5> <Flask: handle page counter information 3.6> <Flask: collect parameters for XSLT translation 3.7> <Flask: process the XSLT translation 3.8> <Flask: save html and process any errors 3.9> return render
Chunk referenced in 3.1

The showpage routine does a similar task to that of my index.py routine in the Apache web servers that I run. It determines the type of file to be served, and in the case of XML files, calls the xsltproc processor to convert them to HTML.

<Flask: check what kind of file is requested 3.4> =
# check what kind of file is requested res=re.match('(.*)\.([^.]+)$',requestedFile) if res: reqtype=res.group(2) if reqtype!='xml': #logMsg("%s: Handling type %s" % (tsstring,reqtype)) reqf=open(requestedFile) result=reqf.read() return result render="" # define the parameters to the translation filename=pagename server='%s:5000' % HouseDefinitions.CENTRAL host=HouseDefinitions.CENTRAL BASE='(not used)' filestat=os.stat(requestedFile) filemod=filestat.st_mtime dtfilemod=datetime.datetime.fromtimestamp(filemod) dtstring=dtfilemod.strftime("%Y%m%d:%H%M") URL="(dummy URL)"
Chunk referenced in 3.3
<Flask: compute the current working directory 3.5> =
# work out the relative current working directory logMsg("requestedFile=%s" % (requestedFile)) res=filepat.match(requestedFile) if res: logMsg("matched") dir=res.group(1) relcwd=dir # protocol for relcwd: # no subdir => relcwd = '' (empty) # exists subdir => relcwd = subdir (no leading or trailing slash) #if dir!="": # requestedFile=dir+'/'+res.group(2) #else: # requestedFile=res.group(2) filename=res.group(2) else: logMsg("not matched") # not ajh (sub)directory, extract full directory path dir=os.path.dirname(requestedFile) relcwd=dir filename=os.path.basename(requestedFile) #if relcwd[0:3]=='www/': # relcwd=relcwd[4:] logMsg("requestedFile=%s" % (requestedFile)) logMsg("relcwd=%s" % (relcwd))
Chunk referenced in 3.3
<Flask: handle page counter information 3.6> =
#logMsg("relfile = %s" % (relfile)) counterName=re.sub("/~ajh/",'',relfile) counterName=re.sub("^/",'',counterName) extnPattern=re.compile("(.xml)|(.html)") counterName=re.sub(extnPattern,'',counterName) counterName=COUNTERS+'/'+re.sub("/","-",counterName) #logMsg("got counter name = %s" % (counterName)) newCounterStr='<?xml version="1.0"?>\n' newCounterStr+='<counter><value>0</value><date>%s</date></counter>' % todayStr try: counterFile=open(counterName,'r') dom=xml.dom.minidom.parse(counterFile) counterFile.close() except IOError: dom=xml.dom.minidom.parseString(newCounterStr) except xml.parsers.expat.ExpatError: dom=xml.dom.minidom.parseString(newCounterStr) except: logMsg("Unexpected error:", sys.exc_info()[0]) raise # now extract count field and update it countNode=dom.getElementsByTagName('value')[0] if countNode.nodeType == xml.dom.Node.ELEMENT_NODE: textNode=countNode.firstChild if textNode.nodeType == xml.dom.Node.TEXT_NODE: text=textNode.nodeValue.strip() countVal=int(text) countVal=countVal+1 textNode.nodeValue="%d" % (countVal) countDate='(unknown)' countNode=dom.getElementsByTagName('date')[0] if countNode.nodeType == xml.dom.Node.ELEMENT_NODE: textNode=countNode.firstChild if textNode.nodeType == xml.dom.Node.TEXT_NODE: countDate=textNode.nodeValue.strip() # write updated counter document try: counterFile=open(counterName,'w') except IOError: counterName='/home/ajh/local/localhost/counters/index' counterFile=open(counterName,'w') domString=dom.toxml() counterFile.write(domString) counterFile.close() #logMsg("domString=%s" % (domString))
Chunk referenced in 3.3
<Flask: collect parameters for XSLT translation 3.7> =
parms="" parms+="--param xmltime \"'%s'\" " % (dtstring) parms+="--param htmltime \"'%s'\" " % (tsstring) parms+="--param filename \"'%s'\" " % (filename) parms+="--param relcwd \"'%s'\" " % (relcwd) parms+="--param URL \"'%s'\" " % (URL) parms+="--param today \"'%s'\" " % (todayStr) parms+="--param host \"'%s'\" " % (host) parms+="--param server \"'%s'\" " % (server) parms+="--param base \"'%s'\" " % (BASE) parms+="--param debug \"'%s'\" " % (debug) logMsg(parms)
Chunk referenced in 3.3
<Flask: process the XSLT translation 3.8> =
# start a pipe to process the XSLT translation # first see if there is a stylesheet rfile=open(requestedFile,'r') i=5 xsltrans="ajhwebpage.xsl" for l in rfile.readlines(): #logMsg(l) if i<=0: break res=re.match('.*xml-stylesheet.*file://.*lib/xsl/(.*)"',l) if res: xsltrans=res.group(1) break i-=1 rfile.close() xslfile="/home/ajh/www/lib/xsl/%s" % (xsltrans) cmd=XSLTPROC+" --xinclude %s%s %s " % (parms,xslfile,requestedFile) #render+="<p>%s</p>" % (cmd) #(pipein,pipeout,pipeerr)=os.popen3(cmd) # # This is the crucial XML -> HTML conversion process! # pid=Popen(cmd,shell=True,stdout=PIPE,stderr=PIPE,close_fds=True) # # now handle what the conversion has done # (pipeout,pipeerr)=(pid.stdout,pid.stderr) if debug: cwd=os.getcwd() logMsg("%s: (cwd:%s) %s" % (tsstring,cwd,cmd)) #sys.stderr.write("(cwd:%s) %s: %s\n" % (cwd,tsstring,cmd)) # report the fact, and the context (debugging purposes) logMsg("%s: converting %s with %s\n" % (tsstring,requestedFile,xslfile))
Chunk referenced in 3.3
<Flask: save html and process any errors 3.9> =
# open a file to save the HTML for debug purposes htmlfilename="/tmp/last.html" htmlf=open(htmlfilename,'w') for line in pid.stdout.readlines(): render+=line htmlf.write(line) htmlf.close() errs=[] for line in pipeerr.readlines(): errs.append(line) #logMsg(line) #logfile=PRIVATE+'/xmlerror.log' #logfiled=open(logfile,'a') if errs: #logMsg("%s: ERROR IN REQUEST: %s" % (tsstring,requestedFile)) #logMsg("<HR/>\n") logMsg("%s: MESSAGES GENERATED BY: %s" % (tsstring,requestedFile)) #logMsg("<PRE>") for errline in errs: #logfiled.write("%s: %s" % (tsstring,errline)) errline=cgi.escape(errline) errline=errline.rstrip() logMsg("%s: %s" % (tsstring,errline)) #logMsg("</PRE>") #logMsg("<p>Please forward these details to ") #logMsg("<a href='mailto:ajh@csse.monash.edu.au'>John Hurst</a>") else: #logfiled.write("%s: NO ERRORS IN %s\n" % (tsstring,requestedFile)) pass #logfiled.close() pipeout.close(); pipeerr.close()
Chunk referenced in 3.3

3.1.3 Start the Flask application

"startFlask.sh" 3.10 =
LOGDIR='/home/ajh/logs/central' HOUSE='/home/ajh/Computers/House' if [ -f ${LOGDIR}/flaskProcess ] ; then for p in `cat ${LOGDIR}/flaskProcess` ; do kill -9 $p done rm ${LOGDIR}/flaskProcess fi ${HOUSE}/lilydaleFlask.py >>${LOGDIR}/flask.log 2>&1 & ps aux | grep "Flask.py" | grep -v grep | awk '{print $2}' >${LOGDIR}/flaskProcess

3.1.4 Watchdog program for Flask

"watchFlask.py" 3.11 =
<edit warning 2.1> time=`/home/ajh/bin/date` curl -s -m 10 http://lilydale.home:5000/house >/dev/null status=$? if [ $status -ne 0 ] ; then echo "$time status is $status, restarting" restart flask else echo "$time status is $status, OK" fi

The flask application seems rather unreliable, and keeps freezing. In an effort to avoid this causing glitches to the access of the HouseMade system, this watchdog program has been implemented. Its operation is to request a download of the house web page, and if this times out, then a restart flask is issued. Essential to its correct operation is that it must be run as root, so that the restart flask operation runs correctly. Hence it must have its sudo sticky bit set, viz sudo chmod 4711 watchFlask.sh - a reminder of this is given by the Makefile (q.v.).

Note that the status returned by curl must be captured, as it is used several times subsequently, and the $? variable gets overwritten.

3.2 The HouseMade module

"HouseMade.sh" 3.12 =
#!/usr/bin/python ## H o u s e M a d e . p y ## ## ********************************************************** ## * do NOT EDIT THIS FILE! * ## * Use $HOME/Computers/House/HouseMade.xlp instead * ## ********************************************************** import datetime import cgi,math,string import re import time import xmlrpclib import ChookDoor import currentState import wx200 from HouseDefinitions import * WServer='http://%s:5000/weather' % (CENTRAL) <HouseMade: define the Generate Weather Data routine 3.13> <HouseMade: define the Generate Solar Data routine 3.14> <HouseMade: define the Generate Tank Data section 3.15> def house(remadr,server,args): import os,sys DEBUG=False SServer='http://%s:5000/solar' % (CENTRAL) TServer='http://%s:5000/tank' % (CENTRAL) WServer='http://%s:5000/weather' % (CENTRAL) <HouseMade: collect date and time data 3.16> picserver='wolseley' HServer='http://%s:5000/heating' % (CENTRAL) MServer='http://%s:5000/house' % (CENTRAL) <HouseMade: check client connection 3.17> (aimtemp,onoff) = (ThermostatSetting,'off') #garedelyon.getHeating() res=0 onoffcolor='blue' # legacy code - will be reinstated some day if onoff=='on': onoffcolor='red' sys.path.append('/home/ajh/Computers/House/') # now respond to any argument requests - can only do if RPC server present active=False if args and CentralGood: currState=Central.getState() for relay in RelayNames: if args.has_key(relay): active=True bitNo=RelayTable[relay] newState=args[relay][0] if newState in ['off','on']: doWhat={'off':Central.setBitOff,'on':Central.setBitOn} change=doWhat[newState](bitNo) print "change bit %d(%s) to %s" % (bitNo,relay,newState) else: # start timer with time == newstate timerCount=int(newState) Central.start(bitNo,timerCount) print "timer started for bit %d(%s) for %d" % \ (bitNo,relay,timerCount) now=datetime.datetime.now() jobtime=str(now-starttime) maxmintab={}#garedelyon.maxminTemp() now=datetime.datetime.now() jobtime=str(now-starttime) import ChookDoor chooks=ChookDoor.ChookDoor() chooks.load() localrise=chooks.sunrise localset=chooks.sunset gateopentime=chooks.dooropen gateshuttime=chooks.doorshut suns = ''' <table> <tr> <td align="right">Today sunrise is at </td><td style="color:#e04000"> %s </td> <td align="right">and sunset is at </td><td style="color:#e04000"> %s </td> </tr> <tr> <td>The chook gate opening time is </td><td style="color:#c04000"> %s </td> <td> and the shutting time is </td><td style="color:#c04000"> %s </td> </tr> </table> ''' % (localrise,localset,gateopentime,gateshuttime) import os,string # determine what relays are currently switched on if CentralGood: state=Central.getState() currentcircuits=[] for i in range(NumberOfRelays): if state[i]==1: currentcircuits.append(RelayNames[i]) if len(currentcircuits) > 1: currentcircuits = "the " + string.join(currentcircuits[:-1], ", ") + " and " + currentcircuits[-1] elif len(currentcircuits) > 0: currentcircuits = "the " + currentcircuits[0] else: currentcircuits = "no" relayState=''' <p> Currently %(currentcircuits)s circuits are on. Visit the <a href="%(HServer)s">Heating Timer</a> page; <a href="%(WServer)s">Weather</a>; <a href="%(SServer)s">Solar Power</a>; <a href="%(TServer)s">Tank Storage</a>. </p> ''' % vars() else: relayState=''' <p> Currently no relay information is available - have you started the relay server? </p> ''' ################### # make the adjust temperature button panel # this is dynamically constructed to show the current aiming temperature. buttonColours=['blue','#10e','#20d','#40b','#609','#807', '#a05','#b04','#c03','#d02','#e01','red'] aimIndex=math.trunc(aimtemp+0.5)-12 if aimtemp<=12.5: aimIndex=0 elif aimtemp>=22.5: aimIndex=11 buttonColours[aimIndex]='yellow' adjustPanel=''' <td><button name="button" value="" type="submit"></button></td> <td bgcolor="%s"> <button name="button" value="cooler" type="submit">COOLER</button> </td> <td bgcolor="%s"><button name="button" value="13" type="submit">13C</button></td> <td bgcolor="%s"><button name="button" value="14" type="submit">14C</button></td> <td bgcolor="%s"><button name="button" value="15" type="submit">15C</button></td> <td bgcolor="%s"><button name="button" value="16" type="submit">16C</button></td> <td bgcolor="%s"><button name="button" value="17" type="submit">17C</button></td> <td bgcolor="%s"><button name="button" value="18" type="submit">18C</button></td> <td bgcolor="%s"><button name="button" value="19" type="submit">19C</button></td> <td bgcolor="%s"><button name="button" value="20" type="submit">20C</button></td> <td bgcolor="%s"><button name="button" value="21" type="submit">21C</button></td> <td bgcolor="%s"><button name="button" value="22" type="submit">22C</button></td> <td bgcolor="%s"> <button name="button" value="hotter" type="submit">HOTTER</button> </td> ''' % (tuple(buttonColours)) ################### Relay Control if CentralGood: row="<tr><th>Name</th><th>bit No</th><th>On/Off</th>" row+="<th>Timer</th><th colspan='8'>Run For</th></tr>" for key in sorted(RelayTable, key=RelayTable.get): bitNo=RelayTable[key] thisBit=Central.getState()[bitNo] thisState=['off','on'][thisBit] newState=['on','off'][thisBit] thisColour=['lightblue','red'][thisBit] row += '<tr bgcolor="%s">\n' % (thisColour) row += ' <td>%s</td>' % (key) row += ' <td bgcolor="%s">%s</td>' % (thisColour,bitNo) # bit number row += ' <td bgcolor="%s">' % (thisColour) row += '<button name="%s" value="%s" type="submit">' % (key,newState) row += '%s</td>' % (thisState) timeLeft=Central.getTimer(bitNo) if timeLeft>0: active=True row += ' <td bgcolor="%s" width="50px">%d</td>' % (thisColour,timeLeft) for t in [30,60,120,300,600,1200,1800,3600]: if t<60: buttontxt="%d secs" % (t) elif t>=3600: buttontxt="%d hour" % (t/3600) else: buttontxt="%d min" % (t/60) t1=t if key=='FloodNDrain' and t>60: t1=60; buttontxt="1 min" row += ' <td><button name="%s" value="%d" type="submit">%s</td>' % (key,t1,buttontxt) row += "</tr>\n" ################### Chook Door house=currentState.HouseState() house.load() chookdoorstate=house.get('chookDoorProve') # was chooks.current if chookdoorstate == 'closed': chookdoorcolour="green" else: chookdoorcolour="red" row += '<tr bgcolor="%s">\n' % (chookdoorcolour) row += ' <td>ChookDoor</td>' row += '<td colspan="11" align="center">%s</td>\n' % (chookdoorstate) row += '</tr>' ################### relaycontrol=""" <form action="%(MServer)s" method="get" name="runRelay"> <table border="1"> %(row)s </table> </form> %(relayState)s """ % vars() else: relaycontrol='<p>No relay information available</p>' ################### Water Storage tanksection=tank([]) ################### Solar Power solarsection=solar([]) ################### Climate weathersection=weather([]) ################### if active: redirect="10;URL='%s'" % (MServer) else: redirect="60" housepage=<HouseMade: generate the web page content 3.18> return housepage if __name__=='__main__': house() ## ## The End ##

3.2.1 define the Generate Weather Data routine

<HouseMade: define the Generate Weather Data routine 3.13> =
def weather(args): WServer='http://%s:5000/weather' % (CENTRAL) MServer='http://%s:5000/house' % (CENTRAL) MAXMINFILE='/home/ajh/logs/central/maxmintemps.log' w=wx200.weather() curtemp = w.inside.temp curhumid = w.inside.humidity outtemp = w.outside.temp outhumid = w.outside.humidity starttime=datetime.datetime.now() yesterday=starttime-datetime.timedelta(days=1) yesterday=yesterday.strftime("%Y%m%d") today=starttime.strftime("%Y%m%d") # check maximum and minimum maxminpat='(\d\d\d\d\d\d\d\d)' # date only maxminpat+=' +([0-9.]+)' # maximum temp maxminpat+=' +([0-9:]+)' # maximum temp time maxminpat+=' +([0-9.]+)' # minimum temp maxminpat+=' +([0-9:]+)' # minimum temp time maxminpat=re.compile(maxminpat) f=open(MAXMINFILE,'r') maxmintable={} for l in f.readlines(): #print "read maxmin line of %s" % (l) res=maxminpat.match(l) if res: d=res.group(1) max=float(res.group(2)) maxat=res.group(3) min=float(res.group(4)) minat=res.group(5) maxmintable[d]=(max,maxat,min,minat) else: print "cannot parse %s" % (l) f.close() if maxmintable.has_key(yesterday): (yestermax,yestermaxat,yestermin,yesterminat)=maxmintable[yesterday] else: print "<P>Min/Max temperatures not available for yesterday</P>\n" (yestermax,yestermaxat,yestermin,yesterminat)=(0.0, '00:00', 0.0, '00:00') if maxmintable.has_key(today): (max,maxat,min,minat)=maxmintable[today] else: print "<P>Min/Max temperatures not available for today</P>\n" (max,maxat,min,minat)=(0.0, '00:00', 0.0, '00:00') # get desired temperature house=currentState.HouseState() house.load() aimtemp=house.get('thermostat') onoffcolor="black" weathersection=""" <h2><a href="%(WServer)s">Weather</a></h2> <image src="personal/tempplot.png"/> <form method="POST" action="house.py"> <p> Outside it is %(outtemp).1fC and %(outhumid)d%% humid. It is currently %(curtemp).1fC and %(curhumid)d%% humid inside. <span style="color:%(onoffcolor)s">The heating is aiming for <input type="text" size="6" name="temp" value="%(aimtemp).1f"></input> C.</span> </p> </form> <p> Temperature Extremes on the outside: <table align="center" width="80%%"> <tr><th>Yesterday</th><th>Today</th></tr> <tr> <td align="center">maximum=%(yestermax)s at %(yestermaxat)s</td> <td align="center">maximum=%(max)s at %(maxat)s</td> </tr> <tr> <td align="center">minimum=%(yestermin)s at %(yesterminat)s</td> <td align="center">minimum=%(min)s at %(minat)s</td> </tr> </table> </p> <p><a href="%(MServer)s">Back to House</a></p> """ % vars() return weathersection
Chunk referenced in 3.12

This routine collects up the climate/temperature/heating data and builds a web page to display it. The web page is returned as a string, allowing it to be directly called by the Flask module.

The maxminTemp() call will return an empty dictionary if it cannot find the maximum and minimum temperatures, so we need key ckecks to avoid run time errors.

3.2.2 define the Generate Solar Data routine

<HouseMade: define the Generate Solar Data routine 3.14> =
def solar(args): SServer='http://%s:5000/solar' % (CENTRAL) MServer='http://%s:5000/house' % (CENTRAL) in_Ah = Central.getSolar(20) out_Ah = Central.getSolar(24) solaramps = Central.getSolar(32) solarbatteryvolts = Central.getSolar(34)*0.1+15 solarpower = solaramps*solarbatteryvolts percentsolar = solaramps * 100.0 / 50.0 in_whr = int(in_Ah*27.6) in_MJ = (in_Ah*27.6*3.6/1000) solarsection=""" <h2><a href="%(SServer)s">Solar power</a></h2> <image src="personal/solarplot.png"/> <p> We are getting %(solaramps).1fA into our %(solarbatteryvolts).1fV batteries for a total power output of %(solarpower).1fW, or about %(percentsolar).1f%% of maximum rated power.<br/> Today we've had %(in_Ah)dAh in (%(in_whr)dWhr=%(in_MJ).1fMJ) and %(out_Ah)dAh out. Note that the ampere-hours out is quoted for the 24vdc level, not the 240vac being generated by the inverter.<br/> There is a <a href="solar.py">detailed log</a> available, and the <a href="https://engage.efergy.com/dashboard" target="_blank"> Engage platform</a> gives real time power consumption. </p> <p><a href="%(MServer)s">Back to House</a></p> """ % vars() return solarsection
Chunk referenced in 3.12

Collect the solar power information for display and generate the related text.

3.2.3 define the Generate Tank Data routine

<HouseMade: define the Generate Tank Data section 3.15> =
def tank(args): TServer='http://%s:5000/tank' % (CENTRAL) MServer='http://%s:5000/house' % (CENTRAL) # get tank data tankvolume=Central.getTank() tanktemp=volts=0.0 tankfull=4500 tankpercent = (tankvolume/float(tankfull))*100 tanksection=""" <h2><a href="%(TServer)s">Tank Storage</a></h2> <image src="personal/tankplot.png"/> <p> The rain water tank is currently at %(tankvolume).1fl(/%(tankfull)dl) = %(tankpercent).1f%%. Check the 7 day graph: </p> <image src="personal/tankplot7.png"/> <p><a href="%(MServer)s">Back to House</a></p> """ % vars() return tanksection
Chunk referenced in 3.12

This section now installed on lilydale.

3.2.4 Collect Date and Time Data

<HouseMade: collect date and time data 3.16> =
# collect date and time data (year, month, day, hour, minute, second, weekday, yday, DST) = time.localtime(time.time()) #tm = time.asctime(time.localtime(time.time())) + \ # ["", "(Daylight savings)"][DST] tm=datetime.datetime.now() tm=tm.strftime("%a, %d %b %Y, %H:%M:%S") starttime=datetime.datetime.now() now=datetime.datetime.now() jobtime=str(now-starttime) # isoweek is Mon-Sun, 1-7, but want 0-origin starting with Sun # Sun Mon Tue Wed Thu Fri Sat # iso: 7 1 2 3 4 5 6 # 0-org 0 1 2 3 4 5 6 weekday=now.isoweekday() % 7
Chunk referenced in 3.12 3.20

The date and time at which this program is run is useful for logging, so collect it at the start of operations. The variable jobtime is intended to check how intensive use of this code may become.

3.2.5 Check the Client Connection

<HouseMade: check client connection 3.17> =
if os.environ.has_key('SSH_CONNECTION'): clientIP=os.environ['SSH_CONNECTION'] res=re.match('^(\d+\.\d+\.\d+\.\d+).*$',clientIP) if res: clientID=res.group(1) else: clientIP='255.255.255.0' #print os.environ print clientIP res=re.match('10\.0',clientIP) if res: clientOK=True else: res=re.match('130\.194\.69\.41',clientIP) if res: clientOK=True else: clientOK=False sys.stderr.write("ATTEMPT TO ALTER HOUSE SETTINGS\n")
Chunk referenced in 3.12

This page is intended to be world-wide-web accessible, and hence we must establish the credentials of the client. If it is on the local network, no problem, but external users must authenticate (username/passwd) before being allowed to alter any parameters.

The IP address is used in the first instance, as given by the environment variable 'SSH_CONNECTION'. If we can extract the 4-block IP address, well and good, otherwise make it the local mask. IP addresses on the local network (10.0.0.*) are allowed, as is the IP address of my work computer. All others pay money, and their names are taken.

3.2.6 Generate the Web Page Content

<HouseMade: generate the web page content 3.18> =
""" <HTML> <HEAD> <LINK REL="SHORTCUT ICON" HREF="favicon.ico"> <meta http-equiv="Refresh" content="%(redirect)s"> <meta http-equiv="Pragma" content="no-cache"> <TITLE>HouseMade</TITLE> </HEAD> <BODY> <h1> <a href="%(MServer)s"> HouseMade: the Hurst House Heater Helpmate </a> </h1> HouseMade thinks it is currently %(tm)s. You might want to see what rain is <a href="http://mirror.bom.gov.au/products/IDR023.shtml"> happening in melbourne</a>, or the <a href="http://www.bom.gov.au/vic/forecasts/scoresby.shtml"> local forecast</a>. %(suns)s %(relayState)s %(relaycontrol)s %(weathersection)s %(solarsection)s %(tanksection)s </BODY> </HTML> """ % vars()
Chunk referenced in 3.12

This is where the framework of the web page is generated. Most of the content is generated elsewhere as strings to be inserted into this template, hence the global dictionary call on vars at the end, with variable content being added via %s formatting imperatives.

3.2.7 Make the Temperature Panel (not currently used)

<house make temperature panel 3.19> =
# make the adjust temperature button panel # this is dynamically constructed to show the current aiming temperature. buttonColours=['blue','#10e','#20d','#40b','#609','#807', '#a05','#b04','#c03','#d02','#e01','red'] aimIndex=math.trunc(aimtemp+0.5)-12 if aimtemp<=12.5: aimIndex=0 elif aimtemp>=22.5: aimIndex=11 buttonColours[aimIndex]='yellow' adjustPanel=''' <td><button name="button" value="" type="submit"></button></td> <td bgcolor="%s"> <button name="button" value="cooler" type="submit">COOLER</button> </td> <td bgcolor="%s"><button name="button" value="13" type="submit">13C</button></td> <td bgcolor="%s"><button name="button" value="14" type="submit">14C</button></td> <td bgcolor="%s"><button name="button" value="15" type="submit">15C</button></td> <td bgcolor="%s"><button name="button" value="16" type="submit">16C</button></td> <td bgcolor="%s"><button name="button" value="17" type="submit">17C</button></td> <td bgcolor="%s"><button name="button" value="18" type="submit">18C</button></td> <td bgcolor="%s"><button name="button" value="19" type="submit">19C</button></td> <td bgcolor="%s"><button name="button" value="20" type="submit">20C</button></td> <td bgcolor="%s"><button name="button" value="21" type="submit">21C</button></td> <td bgcolor="%s"><button name="button" value="22" type="submit">22C</button></td> <td bgcolor="%s"> <button name="button" value="hotter" type="submit">HOTTER</button> </td> ''' % (tuple(buttonColours))

This code is separated out because of the complexity of loading the colours of each of the demand buttons. Each button gets a graduated colour from blue through to red, except for the currently specified temperature, which is shown with a yellow background. Temperatures are rounded to the nearest integer in order to determine which button is so highlighted. Temperatures above and below the selectable range highlight the HOTTER and COOLER buttons respectively.

There is a slight glitch with the operation of this form, in that when no specific temperature button is selected (such as when a text value of temperature is entered, the first button is selected (which would normally be COOLER). (See <house get calling parameters >.) To avoid this, a dummy blank button (value="") is built in at the start of the table list, and when this blank value is recognized, the text value is used instead.

3.3 The HeatingModule module

This web page provides a user-friendly interace to setting the automatic heating on/off times, and the temperatures over the course of a day (this latter function not yet operational). The week is divided into 7 days, each with its own programme.

Each day can have up to 7 blocks of time, where the start time of the first block is midnight (00:00 hours), and the end time of the last block is the next midnight (24:00 hours). The end time of each block can be altered, and the number of blocks is determined by the block that has end time of 24:00.

On saving, if the last end time is earlier than 24:00, a new block is added (up to 5 blocks). If seven blocks are in use, and the last end time does not end at midnight, the program will be incomplete and the actual behaviour is not defined.

The desired temperature of each time block can be set independently.

"HeatingModule.py" 3.20 =
#! /usr/bin/python ## ********************************************************** ## * do NOT EDIT THIS FILE! * ## * Use $HOME/Computers/House/HouseMade.xlp instead * ## ********************************************************** ## ## 20141113:114917 1.0.0 ajh first version with number ## 20141113:114958 1.0.1 ajh elide start times if narrow column ## 20150722:164226 1.1.0 ajh copied from TimerModule and updated ## import cgi,datetime,math,os,sys,re,time from HouseDefinitions import * DEBUG=False <Web: define the heatingData class 3.21> def heating(logMsg,remadr,args): DEBUG=False active=False <HouseMade: collect date and time data 3.16> environ=os.environ if DEBUG: keys=environ.keys() keys.sort() print "environ:" for key in keys: print " %s:%s" % (key,environ[key]) if DEBUG: keys=args.keys() keys.sort() print "arguments", lastKey='' for key in keys: if key[0:3]!=lastKey: print "\n ", lastKey=key[0:3] print "%s:%s" % (key,args[key]), print "\n\n", # put 2 newlines at end server='Central' #print "server=%s" % (server) clientIP=remadr res=re.match('10\.0',clientIP) if res: clientOK=True else: res=re.match('130\.194\.69\.41',clientIP) if res: clientOK=True else: clientOK=False logMsg("clientOK=%s (%s)" % (clientOK,clientIP)) # create data structures and initialize td=heatingData() # load previously saved data td.load('/home/ajh/Computers/House/heatProgram.dat') <Web: heating: collect parameters and update 3.22> <Web: heating: build widths for web page table 3.23> # build web page redirect='' if active: redirect='''<meta http-equiv="Refresh" content="10;URL='%s'>''' % (HServer) out="<HTML>\n<HEAD>\n" out+=redirect out+='<meta http-equiv="Pragma" content="no-cache">\n' out+='<TITLE>HeatingTimer</TITLE>\n' out+="weekday=%d, now=%s" % (weekday,now) if not clientOK: out += "<P>Sorry, you are not authorized to adjust this table</P>" else: out += '<form action="%s" method="get" name="heating">\n' % (HServer) out += ' <button name="button" value="save">save</button>\n' out += ' <table border="1" width="100%" padding="0">\n' for i in range(7): out += " <tr height='40px'>\n" if i==weekday: dayColour="#8f8" else: dayColour="#fff" out += " <td width='10%%' bgcolor='%s'>%s</td>\n" % (dayColour,td.days[i]) out += " <td><table width='100%' height='100%' border='0' padding='0' cellspacing='0'><tr>\n" for j in range(NTempBlocks): if td.width[i][j]==0: continue out += " <td bgcolor='%s' width='%d%%' height='35px' align='center'>\n" % (td.colour[i][j],td.width[i][j]) out += ' <select name="temp-%d-%d" size="1">\n' % (i,j) for k in range(10,27): selected="" if k==td.temp[i][j]: selected="selected" out += ' <option value="%d" %s>%d</option>\n' % (k,selected,k) out += ' </select>\n' out += " </td>\n" out += " <td> </td>\n" out += " </tr>\n" out += " <tr>\n" for j in range(NTempBlocks): if td.width[i][j]==0: continue out += " <td bgcolor='%s' width='%d%%' height='35px' align='center'>\n" % (td.colour[i][j],td.width[i][j]) out += " <table border='0'>\n" (sh,sm)=td.mins2Hours(td.start[i][j]) (eh,em)=td.mins2Hours(td.end[i][j]) if td.width[i][j]>20: out += ' <tr><th>Start</th><th>End</th></tr>\n' out += ' <tr><td>%02d:%02d</td>\n' % (sh,sm) else: out += ' <tr><th>End</th></tr>\n' out += ' <tr>\n' out += ' <td>\n' out += ' <select name="end-%d-%d" size="1">\n' % (i,j) for k in range(0,25): selected="" if k==eh: selected="selected" out += ' <option value="%02d" %s>%02d</option>\n' % (k,selected,k) out += ' </select>\n' out += ' <select name="endmin-%d-%d" size="1">\n' % (i,j) for k in range(0,12): selected="" if 5*k==em: selected="selected" out += ' <option value="%02d" %s>%02d</option>\n' % (5*k,selected,5*k) out += ' </select>\n' out += ' </td>\n' out += ' </tr>\n' out += ' <tr>\n' out += ' <td>\n' out += ' </td>\n' out += ' </tr>\n' out += " </table>\n" out += " </td>\n" out += " <td> </td>\n" out += " </tr></table></td>\n" out += " </tr>\n" out += " </table>\n" out += '</form>\n' out += '<A HREF="http://%s:5000/house">back to house</A>\n' % (CENTRAL) if clientOK: td.save('/home/ajh/Computers/House/heatProgram.dat') print "--------" return out if __name__=='__main__': heating()

3.3.1 Define the heatingData Class

<Web: define the heatingData class 3.21> =
class heatingData(): def __init__(self): self.days=['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'] self.temp=[[ThermostatSetting for j in range(NTempBlocks)] for i in range(7)] self.start=[[0 for j in range(NTempBlocks)] for i in range(7)] self.end=[[0 for j in range(NTempBlocks)] for i in range(7)] self.width=[[10 for j in range(NTempBlocks)] for i in range(7)] self.colour=[['red' for i in range(NTempBlocks)] for j in range(7)] def mins2Hours(self,m): return (m/60,m%60) def hours2Mins(self,h,m): return 60*h+m def load(self,filename='/home/ajh/Computers/House/heatProgram.dat'): f=open(filename,'r') for i in range(7): day=f.readline() res=re.match('Day (\d)$',day) if res: rd=int(res.group(1)) if rd!=i: print "Could not read data at day %s" % (i) for j in range(NTempBlocks): block=f.readline() res=re.match('(\d) (\d\d)(\d\d)-(\d\d)(\d\d):(\d\d)$',block) if res: n=int(res.group(1)) s=60*int(res.group(2))+int(res.group(3)) e=60*int(res.group(4))+int(res.group(5)) t=int(res.group(6)) self.temp[i][j]=t self.start[i][j]=s self.end[i][j]=e self.colour[i][j]=setColour(self.temp[i][j]) if n!=j: print "Error on block %d on day %d" % (j,i) else: break; if block.strip()!='': blank=f.readline() f.close() def save(self,filename='/home/ajh/Computers/House/heatProgram.dat'): f=open(filename,'w') for i in range(7): f.write("Day %d\n" % (i)) for j in range(NTempBlocks): s=self.start[i][j] sh=s/60; sm=s%60 e=self.end[i][j] eh=e/60; em=e%60 t=self.temp[i][j] f.write("%1d %02d%02d-%02d%02d:%02d\n" % (j,sh,sm,eh,em,t)) f.write("\n") pass f.close()
Chunk referenced in 3.20

This class deals with all the logic needed to load and save the heating data, stored in a separate file. It handles conversion from external stored time data in hours:minutes format, converting it to internally stored minutes only (from the start of the day), and reconverting it back again on saving.

It also provides a few simple conversion routines for switching between the formats.

3.3.2 Collect Parameters and Update

<Web: heating: collect parameters and update 3.22> =
# collect parameters if clientOK: if args: active='True' keys=args.keys() keys.sort() for k in keys: #print "k=%s" % (k) res=re.match('(temp|start|end|endmin)-(\d+)-(\d+)',k) if res: t=res.group(1) d=int(res.group(2)) b=int(res.group(3)) #print "got type=%s, day=%d, block=%d" % (t,d,b) if t=='temp': tt=args[k][0] #print "d=%s, b=%s, temp=%s, args[%s]=%s" % (d,b,tt,k,args[k]) t=setTemperature(args[k][0]) td.temp[d][b]=t td.colour[d][b]=setColour(t) pass # 'start' is never used #if t=='start': # #print "<p>",temp,k,args[k] # start[d][b]=int(args[k][0]) # pass if t=='end': #print "<p>",temp,k,args[k] (h,m)=td.mins2Hours(td.end[d][b]) td.end[d][b]=td.hours2Mins(int(args[k][0]),m) pass if t=='endmin': #print "<p>",temp,k,args[k] (h,m)=td.mins2Hours(td.end[d][b]) if h>=24: m=0 else: m=int(args[k][0]) td.end[d][b]=td.hours2Mins(h,m) pass
Chunk referenced in 3.20

3.3.3 Build Widths for Web Page Table

<Web: heating: build widths for web page table 3.23> =
# build widths for table for i in range(7): dayFinished=False for j in range(NTempBlocks): if j>0: # make unused blocks alternate in temperature if td.start[i][j]==1440: # 1440 is midnight, hence unused if td.temp[i][j-1]==10: td.temp[i][j]=ThermostatSetting else: td.temp[i][j]=10 try: td.start[i][j]=td.end[i][j-1] except IndexError: print "index error in HeatingModule: i=%d, j=%d (start=%s, end=%s)" \ % (i,j,td.start,td.end) if td.start[i][j]>td.end[i][j]: td.end[i][j]=60*24 w=td.end[i][j]-td.start[i][j] if w<0: w=0 if dayFinished: w=0 td.width[i][j]=math.trunc(100*w/1440.0) # percentage width (h,m)=td.mins2Hours(td.end[i][j]) if h==24 or h==0: dayFinished=True #print "got day finished at day=%d, block=%d" % (i,j) pass
Chunk referenced in 3.20
<check client IP address OK 3.24> =
if os.environ.has_key('REMOTE_ADDR'): clientIP=os.environ['REMOTE_ADDR'] else: clientIP='255.255.255.255' print os.environ res=re.match('10\.0',clientIP) if res: clientOK=True else: res=re.match('130\.194\.69\.41',clientIP) if res: clientOK=True else: clientOK=False

This code fragment checks to see if the client IP address is on the local network, and sets clientOK to True if it is, False otherwise.

4. The Weather System

The heart of the weather system is an electronic weather station with RS232 output, that is continously monitored by the garedelyon server. Once a minute, this server logs the current inside and outside temperatures, humidity and dew points. This information is stored in a logfile, temp.log in the directory /logdisk/logs/, and accessed by the heating and web systems.

4.1 The C Weather Monitor Program

(Details to be recorded)

4.2 The Python Interface to the Weather System

This is a simple python module that accesses the monitor progam and makes the data accessible as python structures. There is one substantive class, weather, instances of which entities containing the appropriate data.

The null classes environment, wind, rain, and pressure provide further localization of the data values.

environment
defines values for temperature, humidity, and dew point;
wind
defines values for wind gust, wind gustdirirection, avgerage wind speed, avgdir average wind direction, and wind chill factor.

"wx200.py" 4.1 =
#!/usr/bin/python <edit warning 2.1> import os,string,sys,subprocess class environment: pass class wind: pass class rain: pass class pressure: pass class weather: def __init__(self, hostname = "10.0.0.112"): host=os.getenv('HOST') pgm='/home/ajh/Computers/House/wx200d-1.1/wx200 ' opts='--power --battery --display -a --C --kph --hpa --mm --mm/h' cmd="%s %s -l %s --nounits" % (pgm,hostname,opts) f = os.popen(cmd) line = f.readline() l = map(lambda s:string.strip(s[:-1]), string.split(line, "\t")) self.inside = environment() self.outside = environment() self.inside.temp = float(l[0]) self.inside.humidity = int(l[2]) self.inside.dew = int(l[4]) self.outside.temp = float(l[1]) self.outside.humidity = int(l[3]) self.outside.dew = int(l[5]) self.pressure = pressure() self.pressure.local = int(l[6]) self.pressure.sea = int(l[7]) self.wind = wind() self.wind.gust = float(l[8]) self.wind.gustdir = int(l[9]) self.wind.avg = float(l[10]) self.wind.avgdir = int(l[11]) self.wind.chill = int(l[12]) self.rain = rain() self.rain.rate = int(l[13]) self.rain.daily = int(l[14]) self.rain.total = int(l[15]) if __name__ == '__main__': d = weather() for n in dir(d): a = getattr(d, n) for nn in dir(a): print "%s.%s." % (n, nn), str(getattr(a, nn))

4.3 The Weather Logging Process

This code is run once a minute by the EveryMinute.sh script, and simply outputs the inside temperature, the outside temperature, the inside humidity, the outside humidity, and the outside dew point data to the log file.

"logwx.py" 4.2 =
import datetime import re from wx200 import * STATEFILE='/home/ajh/logs/central/wxState.txt' MAXMINFILE='/home/ajh/logs/central/maxmintemps.log' w=weather() inside=w.inside intemp=inside.temp outside=w.outside wind=w.wind rain=w.rain now=datetime.datetime.now() nowstamp=now.strftime("%Y%m%d:%H%M%S") if intemp==0.0: f=open(STATEFILE,'r') prev=f.readline() res=re.match('\d{8}:\d{6} +(\d+\.\d) +(\d+\.\d)',prev) if res: intemp=float(res.group(1)) outside.temp=float(res.group(2)) else: print "Could not match %s" % (prev) line="%s %5.1f %5.1f" % (nowstamp,intemp,outside.temp) line+=" %5.1f %5.1f" % (inside.humidity,outside.humidity) line+=" %5.1f" % (outside.dew) line+=" %5.1f %5.1f" % (wind.gust,wind.gustdir) line+=" %5.1f %5.1f" % (rain.rate,rain.daily) print line # save current state f=open(STATEFILE,'w') f.write(line+'\n') f.close() # check maximum and minimum maxminpat='(\d\d\d\d\d\d\d\d)' # date only maxminpat+=' +([0-9.]+)' # maximum temp maxminpat+=' +([0-9:]+)' # maximum temp time maxminpat+=' +([0-9.]+)' # minimum temp maxminpat+=' +([0-9:]+)' # minimum temp time maxminpat=re.compile(maxminpat) f=open(MAXMINFILE,'r') maxmintable={} for l in f.readlines(): #print "read maxmin line of %s" % (l) res=maxminpat.match(l) if res: d=res.group(1) max=float(res.group(2)) maxat=res.group(3) min=float(res.group(4)) minat=res.group(5) maxmintable[d]=(max,maxat,min,minat) else: print "cannot parse %s" % (l) f.close() changed=False today=now.strftime("%Y%m%d") time=now.strftime("%H:%M") if maxmintable.has_key(today): (max,maxat,min,minat)=maxmintable[today] else: maxmintable[today]=(outside.temp,time,outside.temp,time) changed=True (max,maxat,min,minat)=maxmintable[today] if outside.temp>max: max=outside.temp maxat=time changed=True elif outside.temp<min: min=outside.temp minat=time changed=True if changed: maxmintable[today]=(max,maxat,min,minat) f=open(MAXMINFILE,'w') keys=maxmintable.keys() keys.sort() for k in keys: (max,maxat,min,minat)=maxmintable[k] f.write("%s %5.1f %s %5.1f %s\n" % (k,max,maxat,min,minat)) #print "%s %5.1f %s %5.1f %s" % (k,max,maxat,min,minat) f.close()

5. The Heating System

5.1 AdjustHeat

This script runs every minute on lilydale, to see if the heating should be adjusted. An entry in EveryMinute.sh invokes this program. It has recently (v1.3.0) been revised back to the original concept of allowing an arbitrary desired temperature to be specified, and it records its decisions in the log file heating.log (which see <EveryMinute.sh 15.1>.

"AdjustHeat.py" 5.1 =
<edit warning 2.1> # this code must run on lilydale import datetime import re import RelayControl import xmlrpclib import HeatingModule import currentState from HouseDefinitions import * import wx200 Debug=False hysteresis=0.25 now=datetime.datetime.now() nowStamp=now.strftime("%Y%m%d:%H%M%S") dayofweek=now.isoweekday() % 7 hd=HeatingModule.heatingData() hd.load('/home/ajh/Computers/House/heatProgram.dat') wx=wx200.weather() currentTemp=wx.inside.temp # check if time to change temp t=hd.temp[dayofweek] s=hd.start[dayofweek] e=hd.end[dayofweek] if Debug: print "t=%s, s=%s" % (t,s) hour=now.hour; min=now.minute hourMin=60*hour+min switch=0; ptemp=0 for i in range(len(t)): smins=s[i]; emins=e[i] if Debug: print smins,hourMin,emins if smins<=hourMin and hourMin<emins: switch=emins ptemp=t[i] break desiredTemp=ptemp (swh,swm)=hd.mins2Hours(switch) if Debug: print "on day %s at time %s, planned temp=%d, \ desired temp=%s, next switch=%02d%02d" % \ (hd.days[dayofweek],now.strftime("%Y%m%d:%H%M"),ptemp,\ desiredTemp,swh,swm) # change heating here, but only if need to change! bitNo=RelayTable['Heating'] state='OK'; change=turn='' if currentTemp<desiredTemp-hysteresis: state="low"; turn='on' elif currentTemp>desiredTemp+hysteresis: state="high"; turn='off' if turn: change=', turn %s' % turn print "%s AdjustHeat: current=%4.1fC, desired=%dC, heating is %s%s" % (nowStamp,currentTemp,desiredTemp,state,change) if turn=='on': # turn heating on here Central.setBitOn(bitNo) pass if turn=='off': # turn heating off here Central.setBitOff(bitNo) pass # save current state of desired temperature house=currentState.HouseState() house.load() house.store('thermostat',desiredTemp) house.save()

Note that the variable hysteresis defines how far from the desired temperature the current temperature must depart before the heating will change state.

A suggested improvement is to check the current state of the heating (via the relay controller) to see whether the heating is currently on or off. If the demanded state is the same as the current state, then there is no need to explicitly call for the setBit operation.

5.2 checkTime.py

The responsibility of this program is to periodically (currently every 5 mins, see cron files) check the programmed temperature, and update the demand heating file on garedelyon.

The location of where it runs determines which server is in control of the heating (currently flinders, but this may change in future).

"checkTime.py" 5.2 =
#! /usr/bin/python <edit warning 2.1> import re import datetime import xmlrpclib now=datetime.datetime.now() dayofweek=now.isoweekday() % 7 time=now.strftime("%H%M") days=['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday'] temp=[[ThermostatSetting for j in range(5)] for i in range(7)] start=[[0 for j in range(5)] for i in range(7)] end=[[0 for j in range(5)] for i in range(7)] # get the current desired temperature garedelyon=xmlrpclib.ServerProxy('http://garedelyon:8001') (desiredTemp,onoff) = garedelyon.getHeating() f=open('/home/ajh/Computers/House/tempProgram.dat','r') for i in range(7): day=f.readline() res=re.match('Day (\d)$',day) if res: rd=int(res.group(1)) if rd!=i: print "Could not read data at day %s" % (i) for j in range(5): block=f.readline() res=re.match('(\d) (\d\d)-(\d\d):(\d\d)$',block) if res: n=int(res.group(1)) start[i][j]=int(res.group(2)) end[i][j]=int(res.group(3)) temp[i][j]=int(res.group(4)) if n!=j: print "Error on block %d on day %d" % (j,i) blank=f.readline() f.close() # check if time to change temp t=temp[dayofweek] s=start[dayofweek] e=end[dayofweek] print "t=%s, s=%s" % (t,s) hour=now.hour; min=now.minute switch=0; ptemp=0 for i in range(len(t)): st=s[i];en=e[i] print st,hour,en if st<=hour and hour<en: switch=en ptemp=t[i] break if hour in s and min<=1: desiredTemp=ptemp print "on day %s at time %s, planned temp=%d, desired temp=%s, next switch=%02d00" % \ (days[dayofweek],now.strftime("%Y%m%d:%H%M"),ptemp,desiredTemp,switch) res=garedelyon.setHeating(desiredTemp,onoff) (realtemp,curonoff)=garedelyon.getHeating() if realtemp!=desiredTemp: msg="AWOOGA! AWOOGA! someting wrong with g.setHeating()!!!" msg+="(newdesiredtemp=%s, actualdesiredtemp=%s" % (realtemp,desiredTemp) print msg

The basic logic of this program is to read the programmed temperature changes from the file tempProgram.dat (set by the flinders cgi script heating.py), and adjust the desired temperature to match the programmed temperature.

The key requirement to this logic is that changes should only be made at the appointed switch time.

There seems to be some sort of race condition in the switching logic. The trailing message has been added to try and identify the circumstances under which the planned temperature and desired temperature disagree.

5.3 TempLog

This script logs the house temperature. It runs every minute to maintain a minute-by-minute log. It must be run on garedelyon, since that is where all logging is collected (the log file temp.log is kept in the directory /logdisk/logs).

"TempLog.py" 5.3 =
<edit warning 2.1> # this code must run on garedelyon import os,string,datetime logfile="/logdisk/logs/temp.log" now=datetime.datetime.now() nowstr=now.strftime("%Y%m%d:%H%M%S") log=open(logfile,'a') hostname='10.0.0.101' cmd="/home/ajh/Computers/House/wx200d-1.1/wx200 %s -l --nounits" % hostname f = os.popen(cmd) line = f.readline() l = map(lambda s:string.strip(s[:-1]), string.split(line, "\t")) intemp=float(l[0]) outtemp=float(l[1]) inhumd=float(l[2]) outhumd=float(l[3]) outdew=float(l[5]) currheat=open('/logdisk/logs/heating','r') ch=currheat.readline() therm=float(ch) onoff=currheat.readline().strip() fmt="%s %5.1f %5.1f %5.1f %5.1f %5.2f %5.1f %s\n" vars=(nowstr,intemp,outtemp,inhumd,outhumd,outdew,therm,onoff) log.write(fmt % vars)

6. The Tank System

Define here those program components concerned (solely) with recording and suppling water tank information.

6.1 Water Tank Logging

The water logging code has been re-written from C to Python, to bring it in line with the rest of the system, and to (hopefully) improve the reliability of the logging. The Python code has been added to the existing code to manage the tank system, namely tank.py

6.2 Start the Tank Logging

"startTankLog.sh" 6.1 =
<edit warning 2.1> LOGDIR='/home/ajh/logs/central' HOUSE='/home/ajh/Computers/House' USB=`getDevice tank` kill -9 `cat ${LOGDIR}/tankProcess` rm ${LOGDIR}/tankProcess python tank.py $USB & ps aux | grep "python tank.py" | grep -v grep | awk '{print $2}' \ >>${LOGDIR}/tankProcess

The tank logging is performed slightly differently from the other logging operations, as the tank level transducer operates in a free-running open loop mode. Approximately once a second it sends a burst of data down its RS232 connection, and so it is necessary to have the logging program running constantly to hear those data bursts. This is the purpose of the tank.py program.

The rest of this code is concerned with logging the process ID of the logging program itself, so that it can be started and stopped reliably, without interference from any previous instance.

6.3 Tank Module Functions

The tank module tank.py incorporates some significant legacy code from Nathan's original Nautilus system design. In particular, the read_tank_state and readraw are Python transliterations from Nathan's C code. It is planned to rewrite this module in the near future using a more object-oriented approach.

The code now handles the logging function directly. When called as a main program, it enters an infinite loop, reading and logging the tank transponder, and output a log message once a minute.

When used as an imported module, tank.py defines a number of constants, and provides functions to access tank data and perform temperature compensation (see compensate).

Note that this code is under review, and will be cleaned up in the near future.

"tank.py" 6.2 =
#!/usr/bin/python <edit warning 2.1> import datetime import getopt import re import string import sys import time import os import usbFind LOGFILENAME="/home/ajh/logs/central/tank.log" maxlitres = 2250 # for one tank minlitres = 0 # actual somewhat more maxcap = 68750 # capacitance reading for full supply level mincap = 11484 # capacitance reading when tank is empty NumberOfTanks=2 if NumberOfTanks>1: maxlitres *= NumberOfTanks minlitres *= NumberOfTanks slope=(maxlitres-minlitres)/float(maxcap-mincap) base=minlitres-slope*mincap def compensate(level,temp,ntanks=NumberOfTanks): ''' returns a temperature compensated tank level (NOT litres)''' level=level+213*(temp-28.2)*(level/73035.0) return level def convert(level): '''returns the (uncompensated) volume corresponding to the capacitance meter output value @level. ''' litres=base+slope*float(level) return litres def read_tank_state(): f=open('/home/ajh/logs/central/tankState','r') l=f.readline() if len(l) <= 1: return l = string.split(l) if len(l) < 3: return (level, temp) = map(int, l[1:3]); return (level, temp) def calibrated(): tankdepth,tanktemp=read_tank_state() bigtankheight=1450 tankdepth = (float(tankdepth-mincap)/float(maxcap-mincap))*bigtankheight volume = compensate(tankdepth,tanktemp) return (volume,tanktemp) def readraw(dev): device=os.open(dev,os.O_RDWR) res='' while not res: res=os.read(device,14) time.sleep(5) os.close(device) return res def main(): (level,temp)=read_tank_state() (volume, temp) = calibrated() format = "capacitance reading = %f, " format += "tank temperature = %f" print format % (level, temp*0.1) uncompensated=convert(level) print "uncompensated volume = %5.1f" % (uncompensated) if __name__ == '__main__': (vals,path)=getopt.getopt(sys.argv[1:],'',[]) usb=usbFind.USBclass() tankDevice=usb.device2port('tank') lastMin=60 while True: rawtank=readraw(tankDevice) res=re.match('^ *(\d+) +(\d+) +(\d+).*$',rawtank) if res: level=int(res.group(1)) volts=int(res.group(3)) now=datetime.datetime.now() thisMin=now.minute now=now.strftime("%Y%m%d:%H%M%S") if thisMin!=lastMin: logfile=open(LOGFILENAME,'a') logfile.write("%s %d %d\n" % (now,level,volts)) logfile.close() #print "%s %d %d" % (now,level,volts) lastMin=thisMin pass # end if pass # end while

The temperature compensation values were worked out from a pair of observations:

This gives a line of slope 16.015038 and origin abscissa of 72583.38 for this tank level, assuming that the raw level indicator (corresponding to the frequency oscillator in the water level converter circuit) is linearly related to temperature.

We further assume that these two values themselves are linearly related as the tank level falls, and they themselves should be linearly adjusted by tank level, to get a generalized compensation calculation.

Further work remains to be done on this aspect of tank logging, in particular, how the temperature and raw level data are retrieved and correlated.

7. The Solar System

This is almost verbatim from the central version, although some modifications have been made to correct what appeared to be the presence of obsolete code in read_register.

This code is responsible for building the solar.log file. It is to be run every minute on garedelyon (c.f. EveryMinute.sh), and makes calls upon the pl60 module (a C program) to read the solar data from the solar controller, register by register (Need a link to the solar controller manual here). The log entry is then printed to standard output for logging, and to the file solarState which records the most recent value.

These are the registers of the pl60:
17 Battery Vmax
20 Charge AH
24 Load AH
32 Input Current
34 Battery Voltage
Note that the values returned by these registers (may) need recalibration. Others not mentioned are not used.

7.1 logsolar.py

"logsolar.py" 7.1 =
#!/usr/bin/python <edit warning 2.1> import cgi,string,os import time,sys from HouseDefinitions import CENTRAL LOGS='/home/ajh/logs/%s/' % CENTRAL SOLARSTATEFILE=LOGS+'solarState' SOLARLOGFILE =LOGS+'solar.log' (year, month, day, hour, minute, second, weekday, yday, DST) = time.localtime(time.time()) import cgi def read_register(i): f = os.popen("/home/ajh/bin/pl60 -r %d" % i, "r") line=f.readline() #print "register %d => %s" % (i,line) out = string.split(string.strip(line))[-1] #print out f.close() return out def int_register(i): return int(read_register(i)) in_Ah = int_register(20) # Charge AH out_Ah = int_register(24) # Load AH solaramps = int_register(32)*0.4 # Input Current solarbatteryvolts = int_register(34)*0.1+15 # Battery Voltage solarpower = solaramps*solarbatteryvolts percentsolar = solaramps * 100.0 / 50.0 in_whr = int(in_Ah*27.6) in_MJ = (in_Ah*27.6*3.6/1000) dt = time.strftime("%Y%m%d:%H%M") stateline="%s %d %d %d %d %d %d" % \ (dt,in_Ah,out_Ah,solaramps,solarbatteryvolts,solarpower,in_whr) # this gets redirected at shell level print stateline f=open(SOLARSTATEFILE,'w') f.write(stateline+'\n') f.close()

There is no need to explicitly start this program running, as it is called once every minute by the EveryMinute.sh cron script.

7.2 solar.py

Provide definitions and access functions for the solar controller.

"solar.py" 7.2 =
#!/usr/bin/python <edit warning 2.1> import cgi import os import string import time def read_register(i): f = os.popen("pl60 -b %d" % i, "r") line=f.readline() out = int(line) f.close() return out def int_register(i): return int(read_register(i)) def float_register(i): val=float(read_register(i)) if i==32: val=0.4*val return val def main(): in_Ah = int_register(20) out_Ah = int_register(24) solaramps = int_register(32)*0.4 solarbatteryvolts = int_register(34)*0.1+15 solarpower = solaramps*solarbatteryvolts percentsolar = solaramps * 100.0 / 43.0 t = time.time() dt = time.strftime("%Y%m%d") tm = time.strftime("%H:%M") WHcost = 0.00013 numPanels = 20 data_Panel = 0.0 print "in_Ah=%4.1f, out_Ah=%4.1f, solaramps=%4.1f" %\ (in_Ah, out_Ah, solaramps) if __name__ == "__main__": main()

Note that this code is intended to be imported as a module to python programs that need access to the solar controller. It may be called as a stand-alone program, when it simply prints key data and exits.

8. The Chook Door

The chook house (described separately in the Chickens page) has a door that is automatically controlled, and opens and shuts in accordance with sunrise and sunset times throughout the year.

important note! The chook door proving system has recently been changed from a micropython board to a Beaglebone Black. Consequently some of the following is out-of-date. It will be updated as soon as possible. In the interim, a breadboarded system is running the proving subsystem, and it is not fully integrated into the rest of the system.

There are three programs in this suite:

  1. ChookDoor.py This code provides a module for managing the chook door operations.
  2. checkChooks.py This code updates the houseState file with the current state of the chook door, by calling on the readChook.py code.
  3. ChookProve.py This code reads the single bit value that proves that the chook door is shut, and writes the current state as a text line containing either 'closed' or 'open' to the file /home/ajh/Computers/House/status. It must be run on the Beaglebone system (auteuil) as root, in order to be able to access the (privileged) GPIO pins.

"ChookDoor.py" 8.1 =
<edit warning 2.1> import currentState import datetime import ephem import getopt import re import sys from HouseDefinitions import * compute=0 version='1.0.0' chookFileName='/home/ajh/Computers/House/suntimes.txt' now=datetime.datetime.now() def parse(pat,line): res=re.match(pat,line) if res: return res.group(1) else: return '' class ChookDoor(): def __init__(self): self.opendelay=60 self.shutdelay=20 self.lastrun='' self.current='open' pass <ChookDoor: load 8.2> <ChookDoor: compute 8.3> <ChookDoor: save 8.4> def main(): chooks=ChookDoor() chooks.load() if compute: chooks.compute() chooks.save() print "chooks lastrun = %s" % (chooks.lastrun) print "now = %s" % (now.strftime("%Y%m%d:%H%M")) print "opendelay = %s" % (chooks.opendelay) print "shutdelay = %s" % (chooks.shutdelay) print "today sunrise = %s" % (chooks.sunrise) print "today sunset = %s" % (chooks.sunset) print "dooropen = %s" % (chooks.dooropen) print "doorshut = %s" % (chooks.doorshut) print "chook door is %s" % (chooks.current) if __name__ == '__main__': (vals,path)=getopt.getopt(sys.argv[1:],'cVn=', ['compute','now=','version']) for (opt,val) in vals: if opt=='-c' or opt=='--compute': compute=1 if opt=='-n' or opt=='--now': now=datetime.datetime.strptime(val,"%Y%m%d:%H%M") if opt=='-V' or opt=='--version': print version sys.exit(0) main()

To use this module, import ChookDoor into your code, and create an instance of the class ChookDoor. Data computed by this class is stored in a persistent form in the file chookFileName='/home/ajh/Computers/House/suntimes.txt'. The following methods and entities are available:

load()
Initialize the instance with data from the persistent record.
compute()
Recompute the data in the persistent record (but don't store it).
save()
Save the persistent data (if necessary).
sunrise
Today's sunrise time.
sunset
Today's sunset time.
dooropen
Today's door opening time.
doorshut
Today's door shutting time.
opendelay
minutes after sunrise that the door is to be opened (default is 60 minutes).
shutdelay
minutes after sunset that the door is to be shut (default is 20 minutes).
current
The current state of the door, either 'open' or 'shut'. (Setting this variable has no effect on the door: it is set only as a side effect of the compute() method.)

If the module is invoked as a main program, it performs a load and print operation, unless the -c option is supplied, when it does a recompute of the ephemeral data (and opens/shuts the door if required).

8.1 ChookDoor: load

<ChookDoor: load 8.2> =
def load(self): try: suntimefile=open(chookFileName,'r') innow=suntimefile.readline() self.lastrun=parse('now += (.*)$',innow) self.now=self.lastrun inopdel=suntimefile.readline() self.opendelay=parse('opendelay += (.*)$',inopdel) inshdel=suntimefile.readline() self.shutdelay=parse('shutdelay += (.*)$',inshdel) inrise=suntimefile.readline() self.sunrise=parse('sunrise += (.*)$',inrise) inset=suntimefile.readline() self.sunset=parse('sunset += (.*)$',inset) inopen=suntimefile.readline() self.dooropen=parse('dooropen += (.*)$',inopen) inshut=suntimefile.readline() self.doorshut=parse('doorshut += (.*)$',inshut) incurrent=suntimefile.readline() self.current=parse('door is +(.*)$',incurrent) suntimefile.close() except IOError: pass pass
Chunk referenced in 8.1

Read the persistent data from the external file. This is to allow queries of the data without necessarily recomputing the ephemeral data.

8.2 ChookDoor: compute

<ChookDoor: compute 8.3> =
def compute(self): <ChookDoor: compute ephemeral values 8.5> self.now=now dooropen=sunrise+datetime.timedelta(0,0,0,0,int(self.opendelay)) doorshut=sunset+datetime.timedelta(0,0,0,0,int(self.shutdelay)) self.dooropen=dooropen.strftime("%Y%m%d:%H%M") self.doorshut=doorshut.strftime("%Y%m%d:%H%M") self.sunrise=sunrise.strftime("%Y%m%d:%H%M") self.sunset=sunset.strftime("%Y%m%d:%H%M") lastState=self.current house=currentState.HouseState() house.load() #print "now=%s, dooropen=%s, doorshut=%s" % (now,dooropen,doorshut) if now<dooropen or now>=doorshut: self.current='shut' if lastState=='open': # shut the door Central.start(1,20) house.store('chookDoor','shut') house.save() elif now>=dooropen and now<doorshut: self.current='open' if lastState=='shut': # open the door Central.start(0,20) house.store('chookDoor','open') house.save() else: print "bad compare: now=%s, dooropen=%s, doorshut=%s" % (now,dooropen,doorshut)
Chunk referenced in 8.1

Check if the current time requires a change in the door status. If so, open or close the door as required.

8.3 ChookDoor: save

<ChookDoor: save 8.4> =
def save(self): suntimefile=open(chookFileName,'w') suntimefile.write("now = %s\n" % (self.now.strftime("%Y%m%d:%H%M"))) # now suntimefile.write("opendelay = %s\n" % (self.opendelay)) # opendelay suntimefile.write("shutdelay = %s\n" % (self.shutdelay)) # shutdelay suntimefile.write("sunrise = %s\n" % (self.sunrise)) # sunrise suntimefile.write("sunset = %s\n" % (self.sunset)) # sunset suntimefile.write("dooropen = %s\n" % (self.dooropen)) # dooropen suntimefile.write("doorshut = %s\n" % (self.doorshut)) # doorshut suntimefile.write("door is %s\n" % (self.current)) # current state suntimefile.close() pass
Chunk referenced in 8.1

8.4 ChookDoor: compute ephemeral values

<ChookDoor: compute ephemeral values 8.5> =
now=datetime.datetime.now() thisday=now.day sun=ephem.Sun() melb=ephem.city('Melbourne') prevsunrise=melb.previous_rising(sun) prevsunset=melb.previous_setting(sun) nextsunrise=melb.next_rising(sun) nextsunset=melb.next_setting(sun) # local time zone adjustment prevsunrise=prevsunrise.datetime()+datetime.timedelta(0,0,0,0,0,10) prevsunset =prevsunset.datetime()+datetime.timedelta(0,0,0,0,0,10) nextsunrise=nextsunrise.datetime()+datetime.timedelta(0,0,0,0,0,10) nextsunset =nextsunset.datetime()+datetime.timedelta(0,0,0,0,0,10) #now compute *today*'s rise/setting if thisday==prevsunrise.day: # use previous sunrise=prevsunrise else: # use next sunrise=nextsunrise if thisday==prevsunset.day: # use previous sunset=prevsunset else: # use next sunset=nextsunset
Chunk referenced in 8.3

Use the Python Ephemeral package to compute sunrise and sunset times for Melbourne. Note that the interfaces call for the previous and next times, not today's times, and hence there is a bit of computation needed to work out today's times, regardless of whether the actual sunrise/sunset time is before or after the current time. The times are all computed in UTC, and hence there is a necessary adjustment for local time. Note that daylight saving time is not taken into account!

8.5 checkChooks: update the current chook door state

"checkChooks.py" 8.6 =
#!/usr/bin/python <edit warning 2.1> import currentState import datetime import xmlrpclib LOGFILE='/home/ajh/logs/central/chookDoor.log' auteuil=xmlrpclib.ServerProxy('http://10.0.0.117:8001') door=auteuil.getDoorState() #print door house=currentState.HouseState() house.load() house.store('chookDoorProve',door) house.save() timestamp=datetime.datetime.now() timestamp=timestamp.strftime("%Y%m%d:%H%M") f=open(LOGFILE,'a') f.write("%s: %s\n" % (timestamp,door)) f.close()

This code runs every minute, and inspects the current state of the chook door as given by the door proving subsystem. It uses this to update the general house state file, houseState, as otherwise maintained by <currentState.py 9.1>

8.6 Chook Door Proving Subsystem

Another attempt has been made to employ a Beaglebone system, this time just to run the hard subsystems. The first piece of hardware to be incorporated is the Chook Door Proving Circuit. This is a completely separate circuit (shared a common ground, though) that uses a separate microswitch to detect when the door is properly closed. The microswitch closes on shutting, and the Beaglebone GPIO pins are used a) to read the microswitch, and b) to light some indicator LEDs to show the door state. Red is used to indicate open, Green for closed.

Important Note: It would make sense to combine these two scripts, and eliminate the extra latency occasioned by having to write the status to a file. However, the need for the first script to run as root might complicate matters somewhat.

"ChookProve.py" 8.7 =
<edit warning 2.1> import Adafruit_BBIO.GPIO as GPIO import time greenpin="P8_11" timegreen=0.1 redpin="P8_12" timered=1 inpin="P8_10" GPIO.setup(redpin,GPIO.OUT) GPIO.setup(greenpin,GPIO.OUT) GPIO.setup(inpin,GPIO.IN) last="unknown" while True: if GPIO.input(inpin): new="open" GPIO.output(greenpin,GPIO.LOW) GPIO.output(redpin,GPIO.HIGH) else: new="closed" GPIO.output(redpin,GPIO.LOW) GPIO.output(greenpin,GPIO.HIGH) time.sleep(timegreen) if new!=last: print(new) f=open('status','w') f.write("%s\n" % new) f.close() last=new

This is a simple program that sets up the GPIO pins, then reads the input to check the door stete, lights the appropriate led, and write a simple status into a file.

8.7 Chook Door Proving Server

"ChookDoorServer.py" 8.8 =
#!/usr/bin/python <edit warning 2.1> import datetime import re import subprocess from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler # Restrict to a particular path. class RequestHandler(SimpleXMLRPCRequestHandler): rpc_paths = ('/RPC2',) # Create server server = SimpleXMLRPCServer(("10.0.0.117", 8001), requestHandler=RequestHandler) server.register_introspection_functions() # Define and Register the getHeating function def getDoorState(): p=open('/home/ajh/Computers/House/status','r') openclosed=p.readline().strip() p.close() return openclosed server.register_function(getDoorState, 'getDoorState') # Run the server's main loop print "Chook Door Server restarts" server.serve_forever()

This is an RPC-based server, based upon the <RelayServer.py 10.2> system, that provides a single procedure getDoorState which returns the current state of the chook door, as measured by the proving circuit.

9. The House Computer

9.1 The Current State Interface

"currentState.py" 9.1 =
#!/usr/bin/python Debug=False STATEFILE='/home/ajh/logs/central/houseState' class HouseState(): def __init__(self): state={} def load(self,fname=STATEFILE): f=open(fname,'r') statedata=f.read() if Debug: print statedata self.state=eval(statedata) f.close() def get(self,name): if self.state.has_key(name): return self.state[name] else: print "invalid name %s" % (name) return None def store(self,name,value): self.state[name]=value def save(self,fname=STATEFILE): fn=fname if Debug: fn+='2' f=open(fn,'w') f.write('{\n') for k in self.state.keys(): if Debug: print "saving %s:%s" % (k,self.state[k]) value=self.state[k] if isinstance(value,str): f.write(" '%s':'%s',\n" % (k,value)) else: f.write(" '%s':%s,\n" % (k,value)) f.write('}\n') f.close() def main(): house=HouseState() house.load() if Debug: print house.get('test') house.save() if __name__=='__main__': main()

The currentState routine provides an easy access mechanism to get the current state of various house values, as computed by the various routines. This is in the form of a module that is to be imported by any routine changing the value of a key variable, and provides persistent storage of that variable, until the next time it may be updated.

9.2 The HouseData Server (obsolete)

Defines an RPC server to provide details of the current house state (e.g., the contents of the heating file, the log files, etc.), and to update data as required..

Most of this code was stolen from the Python Library Reference document, and revised from the SimpleHeatExchanger.

This code has been decommissioned, as the server (garedelyon) is defunct. All of these functions are now available through the main house server (lilydale).

"HouseData.py" 9.2 =
#!/usr/bin/python <edit warning 2.1> import datetime import re import subprocess import tank import solar from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler # Restrict to a particular path. class RequestHandler(SimpleXMLRPCRequestHandler): rpc_paths = ('/RPC2',) # Create server server = SimpleXMLRPCServer(("10.0.0.101", 8001), requestHandler=RequestHandler) server.register_introspection_functions() # define some crucial patterns # first, for parsing the temperature (weather station) log file: temppat='(\d\d\d\d\d\d\d\d:\d\d\d\d\d\d)' # date and time temppat+=' +([0-9.]+)' # inside temp temppat+=' +([0-9.]+)' # outside temp temppat+=' +([0-9.]+)' # inside humidity temppat+=' +([0-9.]+)' # outside humidity temppat+=' +([0-9.]+)' # outside dew point temppat+=' +([0-9.]+)' # heating thermostat temppat+=' +(.+)$' # heating on/off temppat=re.compile(temppat) maxminpat='(\d\d\d\d\d\d\d\d)' # date only maxminpat+=' +([0-9.]+)' # maximum temp maxminpat+=' +([0-9:]+)' # maximum temp time maxminpat+=' +([0-9.]+)' # minimum temp maxminpat+=' +([0-9:]+)' # minimum temp time maxminpat=re.compile(maxminpat) # Define and Register the getHeating function def getHeating(): p=open('/logdisk/logs/heating','r') temp=float(p.readline()) onoff=p.readline().strip() p.close() return (temp,onoff) server.register_function(getHeating, 'getHeating') # Define and Register the setHeating function def set(temp,onoff): p=open('/logdisk/logs/heating','w') p.write("%5.2f\n%s\n" % (temp,onoff)) p.close() return 'OK' server.register_function(set, 'setHeating') # Define and Register the getSolar function # moved to RelayServer, 20150509:113400 <HouseData define getTemps 9.3> server.register_function(getTemps, 'getTemps') <HouseData define maxminTemp 9.4> server.register_function(maxminTemp, 'maxminTemp') # Run the server's main loop print "HouseData restarts" server.serve_forever()

Note that the water log should be moved from the root directory to the /logdisk/logs directory to be consistent. Also, we should find a way to manage the bound on the size of the log files (getting the maximum is O(n), for example, whereas it should be O(hours)).

9.2.1 HouseData define getTemps

<HouseData define getTemps 9.3> =
# define the get water level function def getTemps(): logfile='/logdisk/logs/temp.log' cmd=['/usr/bin/tail','-1',logfile] pipe=subprocess.Popen(cmd,stdout=subprocess.PIPE) p=pipe.stdout l=p.readline() p.close() print l res=temppat.match(l) if res: intemp=float(res.group(2)) outtemp=float(res.group(3)) else: intemp=0.0 outtemp=20.0 return (intemp,outtemp)
Chunk referenced in 9.2

9.2.2 HouseData define maxminTemp

<HouseData define maxminTemp 9.4> =
# Define the max and min temperature function # returns a table of maxima and minima, computed previously def maxminTemp(): maxmintable={} logfile='/logdisk/logs/maxmins.log' f=open(logfile,'r') for l in f.readlines(): res=maxminpat.match(l) if res: d=res.group(1) max=res.group(2) maxat=res.group(3) min=res.group(4) minat=res.group(5) maxmintable[d]=(max,maxat,min,minat) f.close() return maxmintable
Chunk referenced in 9.2

9.3 The startHouseData.sh script

"startHouseData.sh" 9.5 =
#!/bin/bash <edit warning 2.1> LOGDIR=/logdisk/logs HOUSEPROC=$LOGDIR/houseProcess HOUSEDIR=/home/ajh/Computers/House kill -9 `cat $HOUSEPROC` /usr/bin/python $HOUSEDIR/HouseData.py >> $LOGDIR/housedata.log 2>&1 & ps aux | grep "HouseData.py" | grep -v grep | awk '{print $2}' >$HOUSEPROC

10. The House Data Server and Relay Control System

Currently, the laptop lilydale is used as both relay controller (through an Arduino circuit) and as general data logger and server for the various house functions. (An exception is the Chook Proving system, see section Chook Door Proving system.)

In this table, the relays are numbered left to right on the computer house panel. Bit 0 is the most significant (or leftmost) bit.

Relay Name Function Wire Colour
0
1
2
3 TopUp Top Up tanks from Mains
4 BottomVegBed Bottom Vegetable Bed Brown
5 MiddleVegBed Middle Vegetable Bed Green
6 TopVegBed Top Vegetable Bed Red
7 CarportVegBed Carport Vegetable Bed White
8 RainForest Rain Forest Sprayers White
9 Woo2Plas WooTank to PlasTank
10 FloodNDrain Flood And Drain
11 Heating Heating

Possibilities for the new relays:

10.1 The Relay Controller Code

This is a simple standalone program used by cron jobs to turn on relays at various times (mainly watering) for fixed periods of time. It calls the RelayServer via RPC calls to actually drive the relays, and really serves only as a CLI parameter handler. The relay name, and the time it is to turn on are supplied by two CLI parameters. If other than two parameters (besides the program name) are supplied, a default is used.

"RelayControl.py" 10.1 =
import time import os import sys import xmlrpclib from HouseDefinitions import * def main(device,timeRunning): # get relay bit for device try: bitNo=RelayTable[device] except KeyError: print "bad relay function key %s" % (relayFunction) sys.exit(1) # start the relay for timeRunning seconds (state,ok)=Central.start(bitNo,timeRunning) print "relays set to %s" % (state) pass if __name__ == '__main__': if len(sys.argv)==3: # get relay name device=sys.argv[1] # get relay time timer=int(sys.argv[2]) else: # default if insufficient parameters device='FloodNDrain' timer=20 main(device,timer)

10.2 Relay Server

The relay server runs all the time, offering a RPC interface to the relay driver. The relay state is represented as a n-element list, where each element represents the relay state as an integer 1 (relay on) or 0 (relay off). It can be controlled either by passing a full state list, or by turning individual bits on and off. The latter is to be preferred, to avoid parallel interaction conflicts.

"RelayServer.py" 10.2 =
#!/usr/bin/python import datetime import os import re import solar import sys import subprocess import threading import time import usbFind from SimpleXMLRPCServer import SimpleXMLRPCServer from SimpleXMLRPCServer import SimpleXMLRPCRequestHandler from HouseDefinitions import NumberOfRelays,RelayNames # define the relay controller device usb=usbFind.USBclass() relayDevice=usb.device2port('arduino') relayDevice="/dev/ttyUSB%1d" % (relayDevice) # Restrict to a particular path. class RequestHandler(SimpleXMLRPCRequestHandler): rpc_paths = ('/RPC2',) # Create server # 10.0.0.112 = lilydale server = SimpleXMLRPCServer(("10.0.0.112", 8001), requestHandler=RequestHandler) server.register_introspection_functions() # define the relay state currentState=[0 for i in range(NumberOfRelays)] currentTime =[0 for i in range(NumberOfRelays)] # time on in seconds nonZeroTimes=[] # those relays on for some time redundantChanges=[0 for i in range(NumberOfRelays)] # count idempotent ops # define the relay control channel relayChannel=open(relayDevice,'w') # open the logfile logname="/home/ajh/logs/central/RelayServer.log" logs=open(logname,'a') # define the convert state to string function def strState(state): str='' for i in range(NumberOfRelays): if currentState[i]==0: str += '0' else: str += '1' return str # Define and Register the getTank function <relayserver: define getTank 10.3> server.register_function(getTank, 'getTank') # Define and Register the getTimer function <relayserver: getTimer 10.4> # Define and Register the getState function <relayserver: getState 10.5> # Define and Register the setState function <relayserver: setState 10.6> # Define and Register the setBit function <relayserver: setBit 10.7> # Define and Register the setBitOn function <relayserver: setBitOn 10.8> # Define and Register the setBitOff function <relayserver: setBitOff 10.9> # Define and Register the start function <relayserver: start 10.10> # define the count down timers process <relayserver: countDown 10.11> # Define and Register the getSolar function def getSolar(regNo): now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") reg=solar.float_register(regNo) #logs.write("%s: getSolar(%d)=>%4.1f\n" % (now,regNo,reg)) #logs.flush(); os.fsync(logs.fileno()) return reg server.register_function(getSolar, 'getSolar') # Run the server's main loop now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") logs.write("%s: RelayServer restarts\n" % (now)) logs.flush(); os.fsync(logs.fileno()) counters=countDown() counters.start() server.serve_forever() counters.join() logs.close()

10.2.1 HouseData define getTank

Here we make a stab at applying a temperature compensation to the water level. It assumes that the compensation is linear in both temperature and level, a somewhat bold assumption.

<relayserver: define getTank 10.3> =
# define the get water level function def getTank(): # dynamically import tank, so that we get latest data settings import tank now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") statefile='/home/ajh/logs/central/tankState' p=open(statefile) l=p.readline() p.close() res=re.match('^(\d\d\d\d\d\d\d\d:\d\d\d\d\d\d) +(\d+) ',l) if res: level=int(res.group(2)) else: level=-1 # now temperature compensate the level, and calibrate litres=tank.convert(level) #logs.write("%s getTank()=>%s (from level=%s)\n" % (now,litres,level)) #logs.flush(); os.fsync(logs.fileno()) return litres
Chunk referenced in 10.2

10.2.2 relayserver: getTimer

<relayserver: getTimer 10.4> =
def getTimer(bitNo): remTime=currentTime[bitNo] return remTime server.register_function(getTimer, 'getTimer')
Chunk referenced in 10.2

10.2.3 relayserver: getState

<relayserver: getState 10.5> =
def getState(): return currentState server.register_function(getState, 'getState')
Chunk referenced in 10.2

10.2.4 relayserver: setState

<relayserver: setState 10.6> =
def setState(newState): now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") if len(newState) < NumberOfRelays: errmsg="%s incomplete state %s in call to setState" print errmsg % (now,newState) return (currentState, errmsg % (newState)) for i in range(NumberOfRelays): currentState[i]=newState[i] s=strState(currentState) relayChannel.write(s+'\n') logs.write("%s setState(%s)\n" % (now,s)) logs.flush(); os.fsync(logs.fileno()) return (currentState,"OK") server.register_function(setState, 'setState')
Chunk referenced in 10.2

10.2.5 relayserver: setBit

<relayserver: setBit 10.7> =
def setBit(bitNo,newValue): now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") if bitNo>=NumberOfRelays: errmsg="%s bad bit number %d in call to setBit" print errmsg % (now,bitNo) return (currentState, errmsg) % (now,bitNo) oldState=currentState[bitNo] currentState[bitNo]=newValue s=strState(currentState) relayChannel.write(s+'\n') if oldState!=newValue: stateStr=['Off','On'][newValue] c=redundantChanges[bitNo] r="previous change repeated %d times" % (c) logs.write("%s setBit%s(%d) newstate=%s, %s (%s)\n" % \ (now,stateStr,bitNo,s,RelayNames[bitNo],r)) logs.flush(); os.fsync(logs.fileno()) redundantChanges[bitNo]=0 else: redundantChanges[bitNo]+=1 return (currentState,"OK") server.register_function(setBit, 'setBit')
Chunk referenced in 10.2

This new routine is intended to coalesce the operations of setBitOn and setBitOff, by passing in an extra parameter newValue.

setBit sets the relay control word to its current state, and with bit number bitNo set to newValue. This new word is written to the relay driver, via the relayChannel routine and the Arduino controller.

An additional piece of logic checks to see if this is actually a change of state, and if it is not, avoids logging the superfluous set operation, but rather increments a counter which is output when the bit is actually changed. Note that this only affects logging - the controller is still updated with the new (unchanged) state.

10.2.6 relayserver: setBitOn

<relayserver: setBitOn 10.8> =
def setBitOn(bitNo): return setBit(bitNo,1) server.register_function(setBitOn, 'setBitOn')
Chunk referenced in 10.2

setBitOn sets the relay control word to its current state, and with bit number bitNo set to a 1. This new word is written to the relay driver, via the relayChannel routine and the Arduino controller.

10.2.7 relayserver: setBitOff

<relayserver: setBitOff 10.9> =
def setBitOff(bitNo): return setBit(bitNo,0) server.register_function(setBitOff, 'setBitOff')
Chunk referenced in 10.2

setBitOff sets the relay control word to its current state, and with bit number bitNo set to a 0. This new word is written to the relay driver, via the relayChannel routine and the Arduino controller.

10.2.8 relayserver: start

<relayserver: start 10.10> =
def start(bitNo,timeon): now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") print "%s: Central.start(%d,%4.1f)\n" % (now,bitNo,timeon) if bitNo>=NumberOfRelays: errmsg="%s bad bit number %d in call to start" print errmsg % (now,bitNo) return (currentState, errmsg) % (now,bitNo) currentState[bitNo]=1 s=strState(currentState) logs.write("%s: startTimer(%d,%4.1f), newstate=%s (%s)\n" % (now,bitNo,timeon,s,RelayNames[bitNo])) logs.flush(); os.fsync(logs.fileno()) # design decision: timeon is relative, not absolute currentTime[bitNo]+=timeon if bitNo not in nonZeroTimes: nonZeroTimes.append(bitNo) s=strState(currentState) relayChannel.write(s+'\n') # turning the bit off is taken care of by the countDown process return (currentState,"OK") server.register_function(start, 'start')
Chunk referenced in 10.2

10.2.9 relayserver: countDown

<relayserver: countDown 10.11> =
class countDown(threading.Thread): def __init__(self): threading.Thread.__init__(self) def run(self): while True: if nonZeroTimes: for bitNo in nonZeroTimes: currentTime[bitNo]-=1 if currentTime[bitNo]==0: # turn this bit off and log the fact currentState[bitNo]=0 s=strState(currentState) relayChannel.write(s+'\n') now=datetime.datetime.now().strftime("%Y%m%d:%H%M%S") print "%s: stopTimer(%d), newstate=%s (%s)" % (now,bitNo,s,RelayNames[bitNo]) logs.write("%s: stopTimer(%d), newstate=%s (%s)\n" % (now,bitNo,s,RelayNames[bitNo])) logs.flush(); os.fsync(logs.fileno()) # remove from nonZeroTimes nonZeroTimes.remove(bitNo) time.sleep(1) # sleep until next cycle
Chunk referenced in 10.2

The purpose of auxilary process countDown is to maintain the count of how long each relay is to be held closed, if it was started with a start call. Note that it is possible to also set relays on and off without a time (using the setBitOn and setBitOff RPC calls), in which case the time remaining is shown as 0 (zero). Such calls will not be automatically turned off, since they do not set the list of nonZeroTimes.

Once a second, the process awakes, and decrements any non-zero counts. These non-zero counts are indicated in the list nonZeroTimes as a partial optimization to avoid testing all relay counts, and to avoid ambiguity about which relays are independently turned on.

10.3 Start the Relay Controller

This short script encapsulates all that is necessary to (re)start the relay server. It records the process ID in the file relayProcess so that when it is restarted, any previous invocation is removed properly.

"startRelayServer.sh" 10.12 =
LOGDIR='/home/ajh/logs/central' HOUSE='/home/ajh/Computers/House' BIN=${HOME}/bin if [ -f ${LOGDIR}/relayProcess ] ; then for p in `cat ${LOGDIR}/relayProcess` ; do kill -9 $p done fi rm ${LOGDIR}/relayProcess #/usr/bin/python ${HOUSE}/RelayServer.py `${BIN}/getDevice arduino` >${LOGDIR}/relay.log 2>&1 & # use non-logging version for now /usr/bin/python ${HOUSE}/RelayServer.py `${BIN}/getDevice arduino` >/dev/null 2>&1 & ps aux | grep "RelayServer.py" | grep -v grep | awk '{print $2}' >${LOGDIR}/relayProcess

10.4 The Old Relay Setup

These are the connections for the old I2C setup that used to run off central, and now decomissioned. They are recorded here to aid in back-tracing legacy connections.

0/0 solarwash 1/0 heating
0/1 rainforest 1/1 nc
0/2 seedling 1/2 nc
0/3 alpine 1/3 rainforestdrip
0/4 floodndrain 1/4 nc
0/5 woo2plas 1/5 bypass
0/6 flush 1/6 northseedling
0/7 creek 1/7 topup

10.5 The Arduino Interface

10.5.1 The Arduino Start Script

"startArduino.sh" 10.13 =
source setupMake.sh make upload

Run this script in the /home/ajh/Computers/House/RelayDriver8way directory on bastille. The setup file sets a few key environment variables.

"setupMake.sh" 10.14 =
export ARDUINODIR=/usr/share/arduino export BOARD=atmega328 export SERIALDEV=`/home/ajh/bin/getDevice arduino`

The make upload command should upload the driver program, and make the arduino ready to accept commands from the command line and the RelayControl.py script. To test that this is working, try:

            echo 001000000000 >/dev/ttyUSB0
          
(ensure that the device named is consistent with the device used in the upload script. Due to the dynamic nature of the usb "plug and play" model, this can change easily, especially when the usb connections have been disturbed.)

11. Hardware Issues

11.1 USB Issues

One of the big headaches in getting this system operational has been the mapping of USB devices to the correct hardware ports. This is clearly a long-standing issue, as many experimenters have posted questions about this on the web.

One of the key software tools to aid in this process of finding the correct mapping is the lsusb command under Linux. Here is the output from it on bastille at 20150426:110547 timestamp (reordered to place items in bus/device order):

ajh@bastille /home/ajh/Computers/House 517 $ date;lsusb
20150426:110547
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 2109:2812  
Bus 001 Device 003: ID 2109:2812  
Bus 001 Device 004: ID 0557:2008 ATEN International Co., Ltd UC-232A Serial Port [pl2303]
Bus 001 Device 005: ID 0557:2008 ATEN International Co., Ltd UC-232A Serial Port [pl2303]
Bus 001 Device 006: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 001 Device 007: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC
Bus 001 Device 008: ID 0bc2:2320 Seagate RSS LLC 
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
       

This is my translation of what these actually are:

Bus Port Device Hardware
1 1 BeagleBone server USB hub
1 2 (not in use)
1 3 (not in use)
1 4 /dev/ttyUSB0 solar
1 5 /dev/ttyUSB1 tank
1 6 (not in use) weather
1 7 /dev/ttyUSB2 Arduino
1 8 (not in use)
2 1 BeagleBone client hub

Note that at the moment several devices (some of which are not in use) on bus 1 cannot be resolved, as they use the same brand of USB/RS232 cable connectors, and there appears to be no serial number information. Note that the IDs are the same as well.

Micheal Ludvig's very helpful page at http://hintshop.ludvig.co.nz/show/persistent-names-usb-serial-devices/ shows how to setup persistent names for these USB-serial devices, but unfortunately it relies upon having unique IDs/SerialNumbers for each device on the bus. You will note that the two ATEN devices have the same Vendor/Product pair, and NO serial numbers, so that they cannot be distinguished at this level. I have posted a comment about this, and will revisit it when/if I receive a reply.

Here are the udev rules I have set up in /etc/udev/rules.d/99-usb-serial.rules to handle my devices:

SUBSYSTEM=="tty", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6001", SYMLINK+="arduino"
SUBSYSTEM=="tty", ATTRS{idVendor}=="067b", ATTRS{idProduct}=="2302", SYMLINK+="tank"
        

(20150606:152721) Note: the Seagate disk has been recording disk errors, which occasionally cause the system to hang. Hence the disk has been removed.

11.2 USB Resolution

Aha! I have finally worked out a solution to the problems outlined above. This article in the Linux FAQ provided the key, and I have extended the ideas outlined in that article. The key point is the fact that the port numbers allocated are ordered in the order of the devices plugged into the hub. I am using a 7-port VIA Labs Hub, and by keeping a list of the devices and port numbers, I can use this together with the information returned by udevadm to map devices to /dev/ttyUSB device files. The page Writing udev rules also gives helpful advice.

That mapping is saved in a file /home/ajh/etc/usbPorts.txt, which is generated by the python program usbFind.py.

I have since discovered similar information resides in the directory /dev/serial/by-path/. I record these here for future use, but for now, things are working so I won't change them! Here are the entries in that directory:

        lrwxrwxrwx 1 root root 13 Jul 10 11:59 pci-0000:00:14.0-usb-0:2.4:1.0-port0 -> ../../ttyUSB3
        lrwxrwxrwx 1 root root 13 Jul 10 15:21 pci-0000:00:14.0-usb-0:2.1.2:1.0-port0 -> ../../ttyUSB1
        lrwxrwxrwx 1 root root 13 Jul 11 11:52 pci-0000:00:14.0-usb-0:2.3:1.0-port0 -> ../../ttyUSB0
        lrwxrwxrwx 1 root root 13 Jul 11 16:11 pci-0000:00:14.0-usb-0:2.2:1.0-port0 -> ../../ttyUSB2
      

Now for my mapping file. The columns are hub port number, device name, USB number. Note that the last column is not necessary (use dashes if not known), and is (re)computed by the usbFind.py program.

"usbPorts.txt" 11.1 =
01 - - 02 arduino 1 03 - - 04 - - 05 spare 2 06 tank 0 07 solar 3

Now for the usbFind.py program, which is written as a module which provides a USBclass, with methods to access and update the usbPorts.txt file. If called as a main program, it does exactly that, and regenerates the mapping file from scratch.

A key method is the get routine, which translates a device name into a USB device port. This is utilised in the getDevice program, installed as part of the bin library.

"usbFind.py" 11.2 =
#!/usr/bin/python <edit warning 2.1> import re import subprocess Debug=False PORTFILE='/home/ajh/etc/usbPorts.txt' class USBclass(): def __init__(self): self.maps=[] self.load() <usbFind: USBclass.load 11.3> <usbFind: USBclass.compute 11.4> <usbFind: USBclass.device2port 11.5> <usbFind: USBclass.save 11.6> def main(): usb=USBclass() print usb.maps usb.compute() print usb.maps usb.save() if __name__ == '__main__': main()

11.2.1 usbFind: USBclass.load

<usbFind: USBclass.load 11.3> =
def load(self): plugfile=open(PORTFILE,'r') plugno=1; portno=1 maps=[] for line in plugfile.readlines(): line=line.strip() res=re.match('^(\d\d) +(.{8}) *((\d|-))?$',line) if res: socket=res.group(1) name=res.group(2) usb=res.group(3) if Debug: print socket,name,usb if name[0]!='-': if usb.isdigit(): maps.append((portno,plugno,name,int(usb))) else: maps.append((portno,plugno,name,'-')) plugno+=1 else: if Debug: print "could not match %s" % line portno+=1 plugfile.close() self.maps=maps return
Chunk referenced in 11.2

11.2.2 usbFind: USBclass.compute

<usbFind: USBclass.compute 11.4> =
def compute(self): udevop=[None for i in range(7)] for usb in range(7): cmd='/bin/udevadm info -a -n /dev/ttyUSB%d' % usb cmd=cmd.split(' ') if Debug: print cmd try: udevop[usb]=subprocess.check_output(cmd) except: print "Cannot get info for device /dev/ttyUSB%s" % (usb) usb=0; portno=0; plugno=1 for op in udevop: if Debug: print "looking at usb=%d, portno=%d, plugno=%d" % (usb,portno,plugno) if not op: usb+=1 continue op=op.split('\n') for l in op: l=l.strip() res=re.match('KERNELS=="2-2.(\d)"$',l) if res: plugno=int(res.group(1)) (port,plug,name,u)=self.maps[plugno-1] if Debug: print "found entry %s" % (l) print "plug number %d is the %s and maps to /dev/ttyUSB%d" % \ (plugno,name,usb) self.maps[plugno-1]=(port,plugno,name,usb) portno+=1 usb+=1 return
Chunk referenced in 11.2

11.2.3 usbFind: USBclass.device2port

<usbFind: USBclass.device2port 11.5> =
def device2port(self,dev): nlen=len(dev) for i in range(len(self.maps)): entry=self.maps[i][2][0:nlen] if dev==entry: return self.maps[i][3] get=device2port # alias
Chunk referenced in 11.2

11.2.4 usbFind: USBclass.save

<usbFind: USBclass.save 11.6> =
def save(self): portsfile=open(PORTFILE,'w') j=0 for i in range(7): #print i+1,self.maps[j] if self.maps[j][0]==i+1: (port,plug,name,usb)=self.maps[j] portsfile.write("%02d %8s%1s\n" % (i,name,usb)) j+=1 else: portsfile.write("%02d - -\n" % i) portsfile.close() return
Chunk referenced in 11.2
"getDevice" 11.7 =
#!/usr/bin/python <edit warning 2.1> import sys from usbFind import * usb=USBclass() device=sys.argv[1] usbNo=usb.get(device) if usbNo==None: print "Cannot get port for device %s" % (device) else: print "/dev/ttyUSB%1d" % (usbNo)

Call this routine as getDevice arduino, where arduino may be replaced by some other device, as listed in the table in <usbPorts.txt 11.1>

11.3 Current Patchboard Wiring

11.3.1 Cellar Computer Board

TOP ROW
Terminal Left Middle Right
name colour function name colour function name colour function
1 gnd black black
2 white white
3 green green
4 blue blue
5 brown brown
6 gnd black green
cable
gnd black
7 red nc white RainForest
8 yellow green
9 brown heater aux relay coil blue TomatoesHanging
10 white heater indicator brown
11 purple Arduino heater relay
12 -
BOTTOM ROW
Terminal Left Middle Right
name colour function name colour function name colour function
1 I black ChookProve red
cable
gnd black red
cable
2 J white seedling A white
3 H green G green Flush
4 K blue B blue ChookUp
5 brown C brown ChookDown
6 gnd black red
cable
7 D white
8 yellow 9 strand cable
to side veg beds
9 strand cable
to side veg beds
green
9 pink {fn} white CarportVegBed E blue Woo2Plas
10 green MiddleVegBed red TopVegBed F brown FloodNDrain
11 unused gnd black
12 blue brown BottomVegBed

These are all numbered top to bottom in each block.

11.3.2 Cellar Door Board

Terminal name colour cnx function
1 gnd black (white)
2 A white (black)
3 G green flush
4 B blue ChookUp
5 C brown ChookDown
6 gnd black
7 D white (white)
8 green
9 E blue Woo2Plas
10 F brown FloodNDrain
11 I black ChookProve
12 J white Seedling
13 H green
14 K blue

These are all numbered top to bottom in each block.

11.3.3 House Corner Board

Far Left (1) Centre Left (2) Centre Right (3) Far Right (4)
Terminal name colour cnx function name colour cnx function name colour cnx function name colour cnx function
1 J nc H (white) 1.3 gnd 1.9 gnd
2 K 2.2 K 1.2 A 1.10 4.2 A 1.9 +12v
3 H 2.1 spare G nc gnd
4 I 3.11 ChookProve spare B 1.5 ChookUp spare
5 B 3.4 ChookUp spare C 1.6 ChookDown gnd 3.8
6 C 3.3 ChookDown spare gnd blue 3.9 Woo2Plas
7 gnd 3.8 spare D nc spare
8 J 3.12 Seedling spare gnd 1.7 4.5 brown 3.10 FloodNDrain
9 gnd 4.1 spare 4.6 gnd black white
10 A 3.2 spare gnd
11 spare spare I 1.4 spare
12 spare spare N 1.8 spare

These are all numbered top to bottom within each block.

11.3.4 BBQ Board

Terminal name colour cnx function Comment Cable
1 J
2 K
3 H (white) nc
4 I yellow
5 B blue.Chook ChookUp towards sky Chook
Cable
6 C brown.Chook ChookDown towards earth
7 gnd
8 J
9 gnd black.Chook -12v (gnd)
10 A white.Chook +12v
11 unused
12 unused

These are numbered left to right.

11.3.5 Restructure Plan

There are some photos of the current state of play, which I will post here soon. In the meantime, there are plans to restructure the relay wiring, using the following colour coding:

colour circuit
red
blue
white
green
yellow
orange
purple heating
grey
brown

I have to admit, I'm not entirely sure this is the best idea ...

11.3.6 Power Bus

Given the frequency with which alterations are made to the control panel, and the need for power sources without interrupting other devices, it is suggested that the power rails be split along the following lines:

BeagleBone 5v
4-way Relay 5v
8-way Relay 5v
Arduino 5v
Lights 12v
Heater Lamp 12v
Weather Station 12v
Solenoids 25vac

Variations on this are possible - for example, combining the 4- and 8-way relay supplies.

11.4 Current Cat5 Ethernet Wiring

There are n switches around the house, known by a single letter name. These are:

D Desktop (in Study)
F Family Room
J Jolimont: 24 port 100Mb switch in cellar
M Middle Bedroom
P Patch Panel: 24 port. Not strictly a switch!
S Study

11.4.1 Cellar Patch Board

There are two components: a 24 port patch panel, and a 24 port 100Mb switch.

Firstly, the patch panel P:

Now the switch (Netgear 24-port 1000Mbs) J:

Port Name Plugged Purpose
1
2
3
4 Study Wallplate Left J4 uplink
5
6
7
8
9 Master Bed Right J9
10 Master Bed Left
11 not used
12
13 Family Vacuum J13 F switch (F1)
14 Dining Left J14
15 Dining Right J15
16 Lounge J16 not wired (?)
17
18
19
20
21
23 Cellar Trains Left J23
24 Cellar Lilydale J24
Port Name Plugged Purpose
1
2
3
4 Study Wallplate Left P4 uplink
5
6
7
8
9 Master Bed Right P9
10 Master Bed Left P10
11 Family Door Left TV HDMI
12 Family Door Right
13 Family Vacuum P13 F switch
14 Dining Left P14
15 Dining Right P15
16 Lounge P16 not wired (?)
17
18
19
20
21
23 Cellar Trains Left P23
24 Cellar Lilydale P24 CHECK THIS!

Note that most of these are direct connexions to the corresponding patch panel port.

11.4.2 Study

There are two switches located in the study: a Netgear DGS1008D 8-port 1000Mbs (known a S, Study), and a Netgear ProSafe 5-port 1000Mbs (known as D, Desktop)

There are also 2 wall plates, each with 2 ports, known as L1, L2, R1 R2 (Left and Right)

Netgear DGS1008D 8-port 1000Mbs

Port Name Plugged Purpose
1 Desktop Link D1 downlink
2 spare
3 Study Wall Plate L2
4 TV Computer Link to Bed 2, TV Computer (via window!)
5 Laser Printer
6 NAS (unused)
7 Billion Modem
8 Efergy Receiver

Netgear ProSafe 5-port 1000Mbs

Port Name Plugged Purpose
1 Study Switch S1 uplink
2 spare
3 Study Wall Plate R1
4 spare
5 Wolseley

Study Wall Plates

Port Name Plugged Purpose
L1 R2
L2 Cellar Patch Panel P4 S3 downlink
R1 Study Wall Plate
R2 L1

11.4.3 Family

Netgear 8-port 1000Mbs

Port Name Plugged Purpose
1 Family Switch Family Vacuum F1
2 spare
3 spare
4 Dimboola Dimboola
5 Laptop (if used)
6 spare
7 spare
8 spare

12. Test Programs

12.1 Check RPC Operation

The following short fragment of code is intended to check the operation of the RPC mechanisms on both garedelyon and lilydale. It provides the user with one RPC object, o (bastille), which can be used to invoke the RPC interfaces. Several such (information supply only) interfaces are invoked as examples.

Usage is to import this code into an interpretive invocation of python, viz from testRPC import *.

"testRPC.py" 12.1 =
<edit warning 2.1> import xmlrpclib from HouseDefinitions import CENTRAL s=xmlrpclib.ServerProxy('http://%s:8001' % CENTRAL) print "options are:" print " s.getState()" print " s.setState([0,0,...]) # 12-element vector of 0/1" print " s.setBitOn(bitnumber) # bit number is an integer (0-11)" print " s.setBitOff(bitnumber) # bit number is an integer (0-11)" print #print " s.getHeating()" #print " s.setHeating(float,'on'/'off')" print " s.getSolar(n) # n is register number (32 is input amps)" #print " s.getTemps()" #print " s.getTank()" #print " s.maxminTemp()" print print "for example," #print " s.getHeating()=%s" % (s.getHeating()) print " s.getState()=%s" % (s.getState()) print

Note that the data logging operations are not currently available.

13. The Log Files

Here is a summary of all the log files maintained:

RelayServer.log
lilydale:/Users/ajh/logs/RelayServer.log logs activity of the relay controller, recording relay sets and resets.
cron.log
bastille:/home/ajh/Computers/House/cron.log logs behaviour of the AdjustHeat.py program, responsible for adjusting the heating on or off, depending upon the current temperature and the desired demand temperature. (This should probably be moved into the logs directory and renamed to heatadjust.log)
housedata.log
garedelyon:/logdisk/logs/housedata.log logs calls on the garedelyon RPC server, along with some debugging information (which should probably be removed).
maxmins.log
garedelyon:/logdisk/logs/maxmins.log logs the maximum and minimum outside temperatures for the last 8 days.
relay.log
garedelyon:/logdisk/logs/relay.log
solar.log
garedelyon:/logdisk/logs/solar.log
tank.log
garedelyon:/logdisk/logs/tank.log
temp.log
garedelyon:/logdisk/logs/temp.log

14. (Re)Starting the Server

14.1 Introduction

This has been a somewhat fraught area of development. Issues with identifying what USB ports are connected to what devices, Arduino startup issues, and just plain knowing the current state of the system is when starting various services, have made this area less than reliable. However, steady progress has been made, and currently the startup scripts are being moved to utilise the UpStart software used to manage all Ubuntu services.

This is a list of all the processes that need starting:

  1. usb - to ensure that all USB connexions are identified correctly
  2. flask - to provide a web server and house interface
  3. arduino - to control the relay drivers
  4. relayserver - to interface to the Arduino
  5. tank logging - to record tank levels
  6. solar logging - to record insolation
  7. weather station - to provide temperature data
  8. chook door

14.2 Non-Upstart Method (more reliable!)

14.2.1 Sort out USB allocations

The USB allocations change on each reboot, and anytime that a USB plug is removed or added, or the USB hub power cycled. Hence when restarting the house system, it is wise to make sure that all USB allocations are known. A script called usbFind.py checks all 7 ports of the hub, and writes the allocations into the file /home/ajh/etc/usbPorts.txt.

          $ + /home/ajh/Computers/House
          $ usbFind.py
          $ -
        

Check the file /home/ajh/etc/usbPorts.txt. It should look like this:

          00 -       -
          01 arduino 0
          02 -       -
          03 -       -
          04 wx200   1
          05 tank    2
          06 solar   3
        

14.2.2 Start the Flask Server

Starting the flask server is simple:

          $ + /home/ajh/Computers/House
          $ startFlask.sh
          $ -
        

Do not worry if you get a message saying that such-and-such process does not exist. That simply reflects the fact that the previous flask server has been killed in some way (for eaxmple, by a reboot).

14.2.3 Start the Arduino Driver

Usually straightforward.

          $ + /home/ajh/Computers/House/RelayDriver8way
          $ . setupMake.sh
          $ make -f arduino.mk upload
          $ -
        

Check that the USB port used matches that identified in the /home/ajh/etc/usbPorts.txt file.

If this process appears to fail, you can check that the Arduino is working by manually sending it a relay state word:

          echo 001000000000 >`getDevice arduino`
        

You should see the relay state change.

14.2.4 Start the RelayServer

Should be straightforward:

          $ + /home/ajh/Computers/House
          $ startRelayServer.sh
          $ -
        

Again, you can ignore messages about non-existent processes.

14.2.5 Start the Tank Logger

          $ + /home/ajh/Computers/House
          $ startTankLog.sh
          $ -
        

The USB port is automatically determined by the startTankLog.sh script. Ignore missing process messages.

14.2.6 Start the Solar Logger

This script is slightly different, as the regular logging is performed by the EveryMinute.sh script, which runs once a minute. Until the USB device is discovered automatically, you need to check the pl60.c script in /home/ajh/Computers/House/pl60-ubuntu-1.0/ uses the correct USB port. Recompile with the correct value (make pl60) and copy the result into the bin directory, so that EveryMinute uses the correct program.

14.2.7 Start the Weather Station Daemon

The weather station uses a daemon to read the weather station data, and make it available to the user level scripts. The daemon is started with:

          $ + /home/ajh/Computers/House/wx200d-ubuntu-1.1
          $ wx200d -s /dev/ttyUSBn
        

where n is the usb number given by the /home/ajh/etc/usbPorts.txt file. Check that this is working correctly by running

          $ wx200
          $ -
        

which should display a heap of weather information. If it doesn't, try power cycling the weather station itself (pull out the power cable, wait a few seconds, and replace it). You will need to reset the date and time, and units.

14.2.8 Start the Chook Door Server

The chook door server provides an RPC interface to interrogate whether or not the chook door is closed, according to the proving microswitch. This circuit is independent of the actual closing and opening process, and provides a safety check that the door is actually closed, once the close command has been issued, and the door has had time to close. It runs continuously on a BeagleBone server (auteuil), hence:

          auteuil $ python /home/ajh/Computers/House/ChookDoorServer.py
        

Since this program runs continuously, it should be started in a separate terminal window, which is then hidden from view, but left running.

14.2.9 Start the Chook Door Prover

The chook door prover is a small program that runs continuously on a BeagleBone server (auteuil), and monitors the state of the proving microswitch. It drives a pair of leds to indicate the state, and also writes the current state to a file status. It must run as root, hence to start it:

          auteuil $ sudo bash
          auteuil # python /home/ajh/Computers/House/ChookProve.py
        

Again, this program outputs the status everytime it changes, so it should be started in a terminal window which is then minimized and left running.

14.3 The New UpStart Configuration

Like its Unix counterpart init.d, the init directory contains scripts to start and stop each component of the HouseMade system.

Here is a list of the relevant files:

  1. /etc/init/chookDoor.conf
  2. /etc/init/flask.conf
  3. /etc/init/relayserver.conf
  4. /etc/init/usbPorts.conf

The upstart software requires that startup scripts are located in /etc/init. This is where the following startup scripts have currently been placed for testing purposes, but the intention is to move them to a separate user area.

"house.conf" 14.1 =
#!/bin/bash start usb start flask

The different services provided are outlined in the following subsections.

14.3.1 USB Allocation

Run the command

          $ start usbPorts
        

This generates a list of the USB devices found in port order in the attached 7-port USB Hub, and identifying, for each list tuple:

  1. Hub socket number;
  2. USB sequence number;
  3. device name;
  4. USB device
So, for example, this invocation records that the Solar controller is the last USB cable in the hub, is plugged into hub socket number 7, and that it has been allocated to /dev/ttyUSB3.

These values are recorded in the file ~/etc/usbPorts.txt and are used by subsequent programs in the startup sequencing.

"usb.conf" 14.2 =
#!/bin/bash USBCONF='/home/ajh/etc/usbPorts.txt' HOUSE='/home/ajh/Computers/House' PYTHON='/usr/bin/python' COMMAND=$1 case $COMMAND: stop) # nothing to do ;; start | restart) ${PYTHON} ${HOUSE}/usbFind.py ;;

14.3.2 Flask Server

Run the command

          $ start flask
        

It normally generates no output, unless the previous invocation failed in some way.

"flask.conf" 14.3 =
#!/bin/bash LOGDIR='/home/ajh/logs/central' HOUSE='/home/ajh/Computers/House' COMMAND=$1 # first check if anything needs to be stopped if [ -f ${LOGDIR}/flaskProcess ] ; then for p in `cat ${LOGDIR}/flaskProcess` ; do kill -9 $p done rm ${LOGDIR}/flaskProcess fi case $COMMAND: stop) # already done ;; start | restart) ${HOUSE}/lilydaleFlask.py >>${LOGDIR}/flask.log 2>&1 & ps aux | grep "Flask.py" | grep -v grep | \ awk '{print $2}' >${LOGDIR}/flaskProcess ;;

14.3.3 Arduino

Start the Arduino. Best to do this while in the Arduino subdirectory, so cd to that:

+ ~/Computers/House/RelayDriver8way
Once there, you need to know what device port the Arduino is connected. The getDevice arduino script is used to do this, and is invoked by the Arduino loader script
            /home/ajh/Computers/House/RelayDriver8way/startArduino.sh
          
Run this script to ensure the Arduino is correctly loaded (in principle, not necessary as the program is in flash memory), and then check that the Arduino is correctly running by issuing
echo 001000000000 >`getDevice arduino`
This should set the third leftmost relay (unused) if the device is working correctly. If it isn't, try running make monitor, and sending 12-bit sequences at it. Quit with C-A \ y and then try the echo instruction again. If that doesn't work, and reports device busy, then kill -9 `fuser /dev/ttyUSBn` where n is the USB number reported as busy. After this, you are on your own.

No, not quite. If you find that the device is locked, look in directory /run/lock/ for a lock file LCK.. that matches your locked device. Removing this file will generally fix the locked device.

14.3.4 Relay Server

Start the Relay Server (first go back to the House directory, do a - to get there):

startRelayServer.sh

14.3.5 Tank Logging

Start the tankLogging:

startTankLog.sh
Again, the device port needs to be known. This is currently provided by the script /home/ajh/bin/getDevice tank, which is called by the startTankLog.sh script.

14.3.6 Solar Logging

Ensure the solarLogging is running. This requires that the C code routine pl60.c is compiled and placed in the home bin directory (/home/ajh/bin/) so that it can be executed by the solar.py module when called by the EveryMinute.sh script.

14.3.7 Weather Station

The weather code relies upon the weather station daemon, wx200d. Firstly, make sure that the weather station port is correctly assigned:

            $ getDevice wx
            /dev/ttyUSB1
          
(Note that device may be slightly different.) Then check that the device alias is correctly set:
            $ ll /dev/wx200 
            lrwxrwxrwx 1 root root 7 Dec 17 16:43 /dev/wx200 -> ttyUSB1
          
Note that the device linked to should agree with the response from the getDevice wx command. If it is not, it will need to be updated. (Use sudo, and document this at some future stage.)

Once this is in place, start the daemon itself with

            $ /home/ajh/Computers/House/wx200d-1.1/wx200d
          

Check the operation of the weather station by running

            $ /home/ajh/Computers/House/wx200d-1.1/wx200
          
which should give a display of all weather station variables.

14.3.8 Regular Service Activity

15. The Cron Jobs

See the cron.xlp file for the actual code. There is just the single interface to cron: the script EveryMinute.sh which is run once a minute.

"EveryMinute.sh" 15.1 =
#!/bin/bash <edit warning 2.1> # collect all the jobs that need to be run regularly into one place BIN=/home/ajh/bin HOUSE=/home/ajh/Computers/House LOGS=/home/ajh/logs/central PERSONAL=/home/ajh/www/personal HOMESERVER=ajh.co:www export HOST=lilydale TZ='Australia/Melbourne'; export TZ export PYTHONPATH=.:/home/ajh/lib/python:/home/ajh/Computers/python export TERM=vt100 # these all done every minute /usr/bin/python ${HOUSE}/checkChooks.py >>${LOGS}/chook.log 2>&1 /usr/bin/python ${HOUSE}/ChookDoor.py --compute >>${LOGS}/chook.log /usr/bin/python ${HOUSE}/logsolar.py >>${LOGS}/solar.log /usr/bin/python ${HOUSE}/logwx.py >>${LOGS}/temp.log /usr/bin/python ${HOUSE}/AdjustHeat.py >>${LOGS}/heating.log tail -1 ${LOGS}/tank.log >${LOGS}/tankState # these only done every 5 minutes minutes=`/bin/date "+%M"` # get current minute five=`expr $minutes % 5` # get minute mod 5 if [ $five -eq 0 ] ; then #/usr/bin/python ${HOUSE}/AdjustHeat.py >>${LOGS}/heating.log /usr/bin/python ${HOUSE}/tempplot.py /usr/bin/python ${HOUSE}/solarplot.py /usr/bin/python ${HOUSE}/tankplot.py /usr/bin/rsync -auv ${PERSONAL}/tempplot.png ${HOMESERVER}/personal/ /usr/bin/rsync -auv ${PERSONAL}/solarplot.png ${HOMESERVER}/personal/ /usr/bin/rsync -auv ${PERSONAL}/tankplot.png ${HOMESERVER}/personal/ /usr/bin/rsync -auv ${PERSONAL}/tankplot7.png ${HOMESERVER}/personal/ # these two commented out due to some funny root mode incompatibility #/bin/bash ${HOUSE}/checkFlask.sh #${BIN}/watchFlask.sh >>${LOGS}/flaskCheck.log fi

Note that this script gets executed every minute, as the name suggests. However, some tasks do not need such frequent service, and hence the calculation of minutes mod 5 so that the adjustHeating is only run every 5 minutes.

16. The Makefile (separate file)

"Makefile" 1.1 =
default=HouseMade GenFiles = install-HouseMade HouseData.py tanklogging.c CGI = ${HOME}/www/cgi-bin HOUSE = ${HOME}/Computers/House BIN = ${HOME}/bin AUTEUIL = auteuil:${HOME}/Computers/House BASTILLE = bastille:${HOME}/Computers/House LILYDALE = lilydale:${HOME}/Computers/House WEBSOURCE = ${HOME}/www/computing/sources/house XSLLIB = /home/ajh/lib/xsl XSLFILES = $(XSLLIB)/lit2html.xsl $(XSLLIB)/tables2html.xsl include ${HOME}/etc/MakeXLP ${GenFiles}:HouseMade.tangle install: install-flask install-definitions install-house \ install-timer install-startFlask HouseMade.tangle HouseMade.xml: HouseMade.xlp Makefile.xlp cat5cabling.xlp \ WebInterface.xlp KeyDataStructures.xlp \ RelayControl.xlp patchboards.xlp Makefile.xlp xsltproc --xinclude $(XSLLIB)/litprog.xsl HouseMade.xlp >HouseMade.xml touch HouseMade.tangle RelayControl.tangle Makefile.tangle HouseMade.html: HouseMade.xml $(XSLFILES) xsltproc --xinclude $(XSLLIB)/lit2html.xsl HouseMade.xml >HouseMade.html html: HouseMade.html pdf: HouseMade.pdf ############################## # INSTALL LILYDALE CODE # ############################## # install flask modules # keep these arranged alphabetically install-AdjustHeat: HouseMade.tangle scp AdjustHeat.py ${LILYDALE}/ touch install-AdjustHeat install-arduino: RelayControl.tangle scp getArduinoUSB.sh lilydale:/home/ajh/bin/ scp startArduino.sh ${LILYDALE}/ scp setupMake.sh ${LILYDALE}/ touch install-arduino install-chook: HouseMade.tangle scp ChookProve.py ${AUTEUIL}/ scp ChookDoorServer.py ${AUTEUIL}/ install-definitions: HouseMade.tangle scp HouseDefinitions.py ${LILYDALE}/ touch install-definitions install-everyMinute: HouseMade.tangle chmod 755 EveryMinute.sh scp EveryMinute.sh ${LILYDALE}/ touch install-everyMinute install-flask: HouseMade.tangle install-house install-timer install-definitions install-watchFlask chmod 755 lilydaleFlask.py scp lilydaleFlask.py ${LILYDALE}/ touch install-flask install-house: HouseMade.tangle install-definitions scp HouseMade.py ${LILYDALE}/ touch install-house install-html: HouseMade.html if /usr/bin/diff HouseMade.html ${WEBSOURCE}/HouseMade.html; then \ cp HouseMade.html ${WEBSOUCE}/HouseMade.html ;\ fi install-relay: RelayControl.tangle chmod 755 RelayServer.py scp RelayServer.py RelayControl.py ${LILYDALE}/ scp testRPC.py ${LILYDALE}/ touch install-relay install-startFlask: HouseMade.tangle chmod 755 startFlask.sh scp startFlask.sh ${LILYDALE}/ touch install-startFlask install-startRelay: HouseMade.tangle install-relay chmod 755 startRelayServer.sh scp startRelayServer.sh ${LILYDALE}/ touch install-startRelay install-tank: HouseMade.tangle scp tank.py ${LILYDALE}/ scp tankplot.py ${LILYDALE}/ scp startTankLog.sh ${LILYDALE}/ touch install-tank install-timer: HouseMade.tangle scp TimerModule.py ${LILYDALE}/ touch install-timer install-watchFlask: HouseMade.tangle scp watchFlask.py.py ${LILYDALE}/../../bin/ echo "Check sudo sticky bit on watchFlask.py" ssh lilydale ls -l ${LILYDALE}/../../bin/watchFlask.py touch install-watchFlask start-auteuil: install-chook ssh root@auteuil chmod 4711 ${HOUSE}/ChookProve.py ssh auteuil /usr/bin/python ${HOUSE}/ChookProve.py ssh auteuil /usr/bin/python ${HOUSE}/ChookDoorServer.py start-lilydale: start-flask start-relay start-arduino touch start-lilydale start-flask: install-startFlask install-flask ssh lilydale ${HOUSE}/startFlask.sh touch start-flask start-relay: install-startRelay ssh lilydale ${HOUSE}/startRelayServer.sh touch start-relay start-arduino: install-arduino ssh lilydale ${HOUSE}/startArduino.sh touch start-arduino start-tank: install-tank ssh lilydale ${HOUSE}/startTankLog.sh touch start-tank <Makefile: install bastille 1.2> **** Chunk omitted! ########################### # MAKEFILE # ########################### makefile: Makefile.tangle ########################### # LEGACY CODE # ########################### install-flinders: HouseMade.tangle executable rsync -auv house.py timer.py flinders:${CGI}/ rsync -auv checkTime.py flinders:${HOUSE}/ rsync -auv tank.py flinders:${HOME}/lib/python/ rsync -auv tank.py flinders:${HOUSE}/ rsync -auv HouseDefinitions.py flinders:${HOUSE}/ rsync -auv TempPlot.py flinders:${HOUSE}/ touch install-flinders install-wolseley: HouseMade.tangle executable rsync -auv house.py timer.py wolseley:${CGI}/ rsync -auv checkTime.py wolseley:${HOUSE}/ rsync -auv tank.py wolseley:${HOME}/lib/python/ rsync -auv tank.py wolseley:${HOUSE}/ rsync -auv HouseDefinitions.py wolseley:${HOUSE}/ rsync -auv TempPlot.py flinders:${HOUSE}/ touch install-wolseley ###################### # INSTALL BY PROGRAM # ###################### install-tank.py: HouseMade.tangle rsync -auv tank.py ${HOME}/lib/python/ rsync -auv tank.py garedelyon:/home/ajh/lib/python/ rsync -auv tank.py flinders:/home/ajh/lib/python/ rsync -auv tank.py flinders:/home/ajh/Computers/House/ ############################# # END OF INSTALLATION STUFF # ############################# executable: house.py timer.py chmod 755 house.py timer.py all: HouseMade.html clean: litclean -rm $(GenFiles)

16.1 Makefile: install bastille

<Makefile: install bastille 1.2> =
############################## # INSTALL BASTILLE CODE # ############################## # install flask modules # keep these arranged alphabetically install-AdjustHeat: HouseMade.tangle scp AdjustHeat.py ${BASTILLE}/ touch install-AdjustHeat install-arduino: RelayControl.tangle scp getArduinoUSB.sh bastille:/home/ajh/bin/ scp startArduino.sh ${BASTILLE}/ scp setupMake.sh ${BASTILLE}/ touch install-arduino install-definitions: HouseMade.tangle scp HouseDefinitions.py ${BASTILLE}/ touch install-definitions install-everyMinute: HouseMade.tangle chmod 755 EveryMinute.sh scp EveryMinute.sh ${BASTILLE}/ touch install-everyMinute install-flask: HouseMade.tangle install-house install-timer install-definitions chmod 755 bastilleFlask.py scp bastilleFlask.py ${BASTILLE}/ touch install-flask install-house: HouseMade.tangle install-definitions scp HouseMade.py ${BASTILLE}/ touch install-house install-html: HouseMade.html if /usr/bin/diff HouseMade.html ${WEBSOURCE}/HouseMade.html; then \ cp HouseMade.html ${WEBSOUCE}/HouseMade.html ;\ fi install-relay: RelayControl.tangle chmod 755 RelayServer.py scp RelayServer.py RelayControl.py ${BASTILLE}/ scp testRPC.py ${BASTILLE}/ touch install-relay install-startFlask: HouseMade.tangle chmod 755 startFlask.sh scp startFlask.sh ${BASTILLE}/ touch install-startFlask install-startRelay: HouseMade.tangle install-relay chmod 755 startRelayServer.sh scp startRelayServer.sh ${BASTILLE}/ touch install-startRelay install-tank: HouseMade.tangle scp tank.py ${BASTILLE}/ scp tankplot.py ${BASTILLE}/ scp startTankLog.sh ${BASTILLE}/ touch install-tank install-timer: HouseMade.tangle scp TimerModule.py ${BASTILLE}/ touch install-timer start-bastille: start-flask start-relay start-arduino touch start-bastille start-flask: install-startFlask install-flask ssh bastille ${HOUSE}/startFlask.sh touch start-flask start-relay: install-startRelay ssh bastille ${HOUSE}/startRelayServer.sh touch start-relay start-arduino: install-arduino ssh bastille ${HOUSE}/startArduino.sh touch start-arduino start-tank: install-tank ssh bastille ${HOUSE}/startTankLog.sh touch start-tank
Chunk referenced in 1.1

17. Document History

20140108:175940 ajh 0.0 first draft, hacked from initial code development scraped from Nathan's code.
20140118:121534 ajh 0.1 Substantial revisions
20140124:105258 ajh 0.2 More substantial revisions
20140128:185934 ajh 0.3 added relay control section, and started on solar logging
20140129:184058 ajh 0.4 completed solar logging, and working on max/min functionality
20140220:160713 ajh 0.5 extended relay control to 12 relays
20140222:180919 ajh 0.5.1 reshuffle some code, particular wrt moving heating control to lilydale
20140425:104834 ajh 0.5.2 move lilydale functions to ringwood
20140426:173322 ajh 0.5.3 various tweaks to get ringwood working, update Makefile
20140427:161124 ajh 0.5.4 add buttons to control watering
20141123:105122 ajh 0.5.5 make max min table more robust with key ckeck
20141130:162421 ajh 0.6.0 move relay timing functions into RelayServer (ringwood)
20141206:123633 ajh 0.6.1 fix bug about timer not turning off
20141206:181650 ajh 0.6.2 rewrite RelayControl and re-organize RelayServer
20150325:095450 ajh 0.6.3 move ringwood functions back to lilydale
20150405:125114 ajh 1.0.0 Rescoped version for orsay
20150419:151754 ajh 1.0.1 working HouseMade and TimerModule included
20150424:174421 ajh 1.0.2 revise setBit operations, tidy documentation
20150426:113730 ajh 1.1.0 add Hardware Section
20150507:215854 ajh 1.1.1 add solar logging code back
20150510:104116 ajh 1.1.2 revised TimerModule to include timerData class, and migrate to 5 minute programming quantum.
20150531:122305 ajh 1.1.3
20150606:115931 ajh 1.1.4 rescoped for bastille
20150628:215955 ajh 1.1.5 add chook door display
20150711:143257 ajh 1.1.6 revised tank logging
20150712:135643 ajh 1.2.0 Removed several separate files, as they were getting in the way, rather than helping. Revamped everything in terms of referring to the house computer as "central", rather than its actual hostname. This is apropos of the Beaglebone bastille dying, and being replaced by lilydale. Also added the usbFind code and explanations.
20150722:163546 ajh 1.3.0 weather station now operational. Main change is to rewrite the TimerModule into the HeatingModule, but other changes are also underway. The web interface has been updated to show temperature data.
20150726:195116 ajh 1.3.1 setup weather, solar and tank interfaces, allowing separate flask interfaces
20150729:114603 ajh 1.3.2 add the currentState.py routine
20150731:082907 ajh 1.3.3 generalize thermostat initialization
20150816:155707 ajh 1.3.4 add section on UpStart operation
20160131:154105 ajh 1.3.5 add restart of chook door proving
20160204:171508 ajh 1.3.5 add cat5 cabling documentation
<current version 16.1> = 1.3.6
<current date 16.2> = 20160204:171508

18. Indices

18.1 Files

File Name Defined in
AdjustHeat.py 5.1
ChookDoor.py 8.1
ChookDoorServer.py 8.8
ChookProve.py 8.7
EveryMinute.sh 15.1
HeatingModule.py 3.20
HouseData.py 9.2
HouseDefinitions.py 2.2
HouseMade.sh 3.12
Makefile 1.1
RelayControl.py 10.1
RelayServer.py 10.2
TempLog.py 5.3
checkChooks.py 8.6
checkTime.py 5.2
currentState.py 9.1
flask.conf 14.3
getDevice 11.7
house.conf 14.1
lilydaleFlask.py 3.1
logsolar.py 7.1
logwx.py 4.2
setupMake.sh 10.14
solar.py 7.2
startArduino.sh 10.13
startFlask.sh 3.10
startHouseData.sh 9.5
startRelayServer.sh 10.12
startTankLog.sh 6.1
tank.py 6.2
testRPC.py 12.1
usb.conf 14.2
usbFind.py 11.2
usbPorts.txt 11.1
watchFlask.py 3.11
wx200.py 4.1

18.2 Chunks

Chunk Name Defined in Used in
ChookDoor: compute 8.3 8.1
ChookDoor: compute ephemeral values 8.5 8.3
ChookDoor: load 8.2 8.1
ChookDoor: save 8.4 8.1
Flask: check what kind of file is requested 3.4 3.3
Flask: collect parameters for XSLT translation 3.7 3.3
Flask: compute the current working directory 3.5 3.3
Flask: define the showpage routine 3.3 3.1
Flask: define the various flask URLs and calls 3.2 3.1
Flask: handle page counter information 3.6 3.3
Flask: process the XSLT translation 3.8 3.3
Flask: save html and process any errors 3.9 3.3
HouseData define getTemps 9.3 9.2
HouseData define maxminTemp 9.4 9.2
HouseMade: check client connection 3.17 3.12
HouseMade: collect date and time data 3.16 3.12, 3.20
HouseMade: define the Generate Solar Data routine 3.14 3.12
HouseMade: define the Generate Tank Data section 3.15 3.12
HouseMade: define the Generate Weather Data routine 3.13 3.12
HouseMade: generate the web page content 3.18 3.12
Makefile: install bastille 1.2 1.1
Web: define the heatingData class 3.21 3.20
Web: heating: build widths for web page table 3.23 3.20
Web: heating: collect parameters and update 3.22 3.20
check client IP address OK 3.24
current date 16.2
current version 16.1
house make temperature panel 3.19
relayserver: countDown 10.11 10.2
relayserver: define getTank 10.3 10.2
relayserver: getState 10.5 10.2
relayserver: getTimer 10.4 10.2
relayserver: setBit 10.7 10.2
relayserver: setBitOff 10.9 10.2
relayserver: setBitOn 10.8 10.2
relayserver: setState 10.6 10.2
relayserver: start 10.10 10.2
usbFind: USBclass.compute 11.4 11.2
usbFind: USBclass.device2port 11.5 11.2
usbFind: USBclass.load 11.3 11.2
usbFind: USBclass.save 11.6 11.2

18.3 Identifiers

Identifier Defined in Used in
heating 3.20 3.2
house 3.12 3.2
hysteresis 5.1
jobtime 3.16

730 accesses since 21 Feb 2016, HTML cache rendered at 20160210:1949