Recording the Weather

For quite some time, I have been planning to create new software to record data from my weather station. The dalweathdb-2.2 which I have been running for the past (lots of) years is limited in many ways, not in the least that it only really works with 1-Wire sensors.

One of the main issues that I have faced is that of data aggregation; the data structure of dalweathdb-2.2 has no concept of days – every reading for every sensor just gets recorded against a timestamp. The queries and ancillary code required to get daily averages, therefore, can be quite excruciating.

About a year or so ago, I came up with the idea of quite a different data structure, using a MongoDB document  for every day. This document then contains an array of timeslots, each timeslot then containing an array of data for each sensor. Every time a sensor reading is added to a timeslot, aggregate values are calculated and written into an aggregates array. So, to find the mean temperature (for instance) for a given day, no further calculation is required. Plot mean temperatures for a year? Just retrieve the 365 (or 366) documents, and there you are.

This isn’t just talk. OK, some of it is – I have yet to write the aggregation functions – but the bulk of the code has been written, and anyone can grab it and play with it on Github.

The software in question does NOT talk to sensors – it’s a data repository, accessed through a JSON API. My idea is that data is acquired through a network of lightweight devices (Raspberry Pi) talking to a range of 1-Wire, I2C, SPI-connected sensors, then sending data off to the repository for collection/reporting.

My next task involves writing the migration code to convert something like 10 years of legacy data over to the new system. Or maybe I’ll just go and hide under the bed.

FOOTNOTE: If you think that MongoDB is over-the-top for something like this, yes, I did think of that; it’s just the most convenient way I know of storing and retrieving JSON. Also, the fact that it runs on my Macbook Air without even raising a sweat means it’s not that resource-hungry.

Weather: As Above, So Below


I have been measuring/recording the weather for a few years now. I started off with a weather station based on Dallas Semiconductor 1-Wire technology. The original system comprised a number of temperature sensors, a cheap, commercial rain gauge hacked and converted to 1-Wire use, and an AAG wind instrument (out of commission, due to multiple issues.) The 1-Wire data bus connected (and still connects) to my office server via a home-brew 1-Wire to serial adapter, with data being recorded on old, unmaintained software going by the name of dalweathdb. Data is stored in a MySQL database, which is reported as HTML pages. Rain and temperature data are available on a public page on my main blog. A very out-of-date page describes the original system.

There have been a few repairs, although few changes, to the original system other than the decommissioning of the wind instrument and replacement of the rain gauge. [In a stroke of luck, the old gauge started playing up, so I looked for a new on on eBay and was able to obtain the same model of gauge used by the Bureau of Meteorology, decommissioned from an WA emergency services site. Electronics had been removed, but I was able to graft those from my old gauge into it, so can now measure precipitation down to 0.2mm.]


I have, for some time, had plans to expand the system, and update to more usable software. The to-do list looks something like this:

  • Move from a pure 1-Wire system to a more heterogenous system, allowing the use of sensors with I2C, SPI, interfaces.
  • Add internal and external humidity sensors.
  • Add a barometer.
  • Change 1-Wire software to OWW, communicating with a central hub using a web API.
  • Use Raspberry Pi units, talking to the API, to interface I2C, SPI-connected sensors.
  • Change database backend to MongoDB, where a single document is created each day, holding all data for all sensors, along with daily aggregates and statistics. (Greatly simplifies reporting – aggregate 365 records per year, rather than every sensor, every five minutes.)
  • Add field mill to measure cloud charge – doubles as early-warning system for electrical storms.
  • Add earth field magnetometer, based on the same inexpensive devices used in mobile phones, to extend terrestrial weather to SPACE WEATHER!

As Above, So Below

You don’t need to be in space to monitor space weather. Solar emissions interact with the earth’s magnetic field (some people are lucky enough to live at latitudes where auroras can be seen!) which can be detected by us, on the ground, using magnetometers.

Magnetometers are now cheap; a module equipped with a Honeywell HMC5883L 3-axis digital compass integrated circuit can be obtained for as little as $3. Coupled with a microcontroller (also cheap,) this would appear to make an inexpensive earth field magnetometer a feasible addition to a weather, or should I say environmental monitoring, station.

My plan is to use a Freescale Freedom Board (there are a few models,) being an inexpensive microcontroller development board based on the ARM Cortex M0+. My reasons for this choice are:

  • I have had a good relationship with Freescale Semiconductor – very supportive of the small developer,
  • These boards pack a lot of punch for the low price-tag,
  • Easy to use/programme,
  • USB connections – should be able to write a serial data stream to a laptop for rapid development,
  • Very good programming training resource at MCU on Eclipse. (The author is also very responsive and helpful to questions.)


Building a system to capture and record magnetometer data does not present too much of a challenge. However, there are certain questions that need to be addressed:

  • Sample rate. At what rate should samples be recorded? (I would incorporate averaging/noise cancellation into the capture process, so report on every 1024 polls of the sensor, or whatever.)
  • Position. Can this sit on the ground, should it be buried, and, if so, how deep?
  • Alignment. Would assume that Z-axis should be perpendicular to the ground (the accelerometer on the Freedom board should help with this,) but which way should X/Y point – magnetic north, true north, …?
  • Calibration. How do we do this, or can we work in relative units?

If you have answers to these, please comment!

Spread The Love

Using standard/readily available modules should allow anyone so wishing to duplicate the hardware. Software can go in Github. How about a global earth field magnetometry network?


The hardware required is cheap and readily available, and can be easily substituted. (Could reproduced using Arduino, for instance.) This project appears to be feasible – comments most welcome.


UPDATE 2014-10-07 – I have now written some software to replace my old weather station software, the data format being deliberately flexible in order to accommodate new sensors, such as a magnetometer.

Trouble-shooting The Great Divider

Today I put the Great Divider to the ultimate test – running it up to 90kV. Having managed to destroy the transformer part of the 90kV power supply when doing tests yesterday, I hooked the voltage multiplier (the green block with the tube coming out of it, in the photograph) up to another transformer and a better driver.
testing at 90kVWith the 5kV electrostatic voltmeter (the instrument in the wooden case) connected to the x25 tap (labelled like that so I know to multiply the meter reading by 25,) I slowly wound up the input voltage to 26V, until the meter read 3.6kV, representing an output voltage of 90kV.

Whilst there was a fair bit of hissing, ozone, and a few sparks, these were all around the voltage multiplier – the anti-corona treatment of the Great Divider appearing to perform its task.

When I moved on to test some other transformers, however, the meter wasn’t moving on the x5 tap, even slightly. Suspecting that – for no reason I could think of – one of the resistors had failed at way below its rated voltage (3.5kV,) I stripped the instrument down for inspection and testing.

trouble-shooting the Great DividerI performed the same set of tests as I did before final assembly, using a voltage reference (grey box in photograph,) and high-impedance buffer (green circuit board) built for the purpose. (The creation of dependent instruments was one of the reasons the Great Divider took so long to complete.)

To my great puzzlement, all tests were successful. There appeared to be nothing wrong whatsoever, and I still can’t for the life of me figure out what was going wrong earlier. The meter itself appeared to be working, so why it should fail to move on the x5 tap, I have still to determine. At least, however, I got the opportunity to once again be impressed by the accuracy of the x25 and x50 taps – and to know that I have the ability to measure high voltages with a fair degree of accuracy.


For those who like numbers, the transformer of the 90kV supply was being driven with a 27.4V signal at 15.6kHZ, with a duty cycle of 77.7%. Input current was 2.4A, so total input power was just under 66W. With the transformer unloaded (voltage multiplier disconnected,) the secondary voltage read approximately 4.7kV, and the input current was 1.2A.

Building The Great Divider

completed voltage dividerHow do I measure very high voltages? I have previously written about an electrostatic voltmeter that measures up to 500V, and have another that that measures up to 5000V (5kV.) What if I want to 20kV, or 100kV? A single instrument that can measure such a voltage directly may not be very useful when measuring much smaller voltages. Rather than acquire lots of different instruments to handle different voltage ranges, the approach I am taking is to use one or two instruments that measure relatively low voltages, and use a voltage divider to extend their range.

Some time ago, I acquired an ex-equipment power supply, which supposedly has an output voltage of 90kV. I built a driver circuit for it and fired it up – the way pieces of dust and small objects flew around my bench told me that, yes, it was certainly producing a substantial voltage, bit I had no way of knowing just how substantial, especially since I wasn’t using the original driver circuit. Fast forward to this year, when I decided to build an instrument to measure high voltages for another project. Since I had the 90kV supply, I thought it appropriate that whatever I built should be able to answer the question, once and for all, as to whether the advertised 90kV really is 90kV.

Originally just called the Big Divider (I onlycame up with Great Divider when writing this article,) my design was a total of fifty 22MΩ resistors, tapped at various points, giving me division ratios of between 1:5 and 1:50. The resistors I selected (a Vishay product, Element 14 part number 190-1890,) have a rated working voltage of 3500V, so the entire string should – in theory – be able to work up to 175kV. Due to issues with insulation and corona discharge, I have no intention of operating the divider much above the 90kV of my supply. (At one point, I considered flooding the divider with sulphur hexafluoride, an insulating gas, but deemed this way to expensive.)

voltage divider circuit board layoutThis image is a capture from the CAD software (Cadsoft Eagle) used to lay out the divider boards. I made five copies of this design on a single sheet, then printed to polyester drafting film, and, using an ultra violet light-box,  contact-printed onto un-etched circuit boardscopper-clad circuit board stock, which is coated with a photosensitive material. After exposure, boards are developed in a sodium metasilicate solution, which removes the exposed photosensitive material.  The developed

etching tankboards (right hand board in picture) are then etched using a solution of ammonium  persulphate. Unfortunately, I ended up having to do this twice. I tried out a cheaper type of board, which uses a phenolic substrate, rather than my usual FR4 fibreglass. When I guillotined the board to divide it into the five boards which constitute the end product, the substrate shattered. With the second batch – made on FR4 – things went as planned, so I drilled holes for the component leads and mounting bolts and proceeded to the next step.

2013-09-14 15.13.28_lznThe resistors used are specified as having a value of 22MΩ, with a tolerance of ±5%. This means that any given resistor could have a value anywhere between 20.9MΩ and 23.1MΩ. In order to get the division ratios as close as possible to the 1:50, 1:25, 1:12.5, etcetera, I had to measure (twice) and record the value of every single resistor. I keyed all this data into a spreadsheet and then went through the painstaking process of ensuring that the lower board in the divider was as close to one-fifth the total resistance as possible, and that each tap on

measuring resistance values

that board was as close to the correct ratio against the total resistance.  Painstaking doesn’t even begin to describe the process but, once I had done all the matching, overall tolerance was better than 1%, so well worth it.

Measured resistors were inserted into a marked sheet of paper so that I could find them again for assembly. The following picture is the last that was seen of the pretty colours of the resistors.

Assembled boards.The boards are mounted on an Acrylic frame using nylon stand-offs and fasteners. With the high voltages involved, I tried to use as little conductive material in the construction as possible.

insulated board

Anticipating issues with corona discharge, and having rules out the use of sulphur hexafluoride as an insulator, I opted to use a spray-on conformal coating (Electrolube DCR modified silicone conformal coating SCC3,) applying several coats. How well this performs is yet to be seen, as the maximum voltage with which I have tested the finished assembly so far was a measly 100V.

Initial testing involved connecting the divider to various reference voltages and measuring the voltage at each tap point. Due to the relatively low input impedance of the multimeter I was using, connection was via a very high impedance op amp buffer (input impedance > 10^12Ω) so as not to distort readings by having a relatively low value resistance connected in parallel. Voltages were recorded in a spreadsheet and the actual ratios calculated. As I can’t be entirely sure how accurate my multimeter is, the errors against the calculated ratios came out to be negligible – the 1:50 measuring as 1:49.91, the 1:12.5 measuring as 1:12.49, etcetera. The effort of resistor matching having paid off, as far as I am concerned. I will be performing further measurements with a voltmeter with greater precision to get more exact ratios – but not until said instrument goes beyond being a bare circuit board, a bag of bits, and unwritten software!

divider with laser-cut case components

The last step in Building the Great Divider was to give it a case. I considered various ways of mounting the acrylic tube, and decided that the best approach would be to design a case and get it laser cut out of acrylic by Ponoko.

I do my laser-cutting design in SVG, using a simple text editor (vim) and a web browser to see what I’m doing – not being a fan of using what is effectively art software for CAD work. I uploaded the SVG file to Ponoko on a Sunday, paid an extra $5 for one-week in-factory handling, and was delighted when it arrived all the way from New Zealand just seven days later. The photograph shows the divider in its tube, sitting on the cut sheet just received from Ponoko.

photo of completed divider

Which brings us to this full-sized image of the Great Divider.

The connectors are 4mm banana jacks in Bakelite – vintage ex-Soviet military. Cables are silicone insulated EHT cable, rated to 20kV (so it doesn’t matter that some are touching, down below.) To reduce corona discharge from the terminals, they have been insulated with clear nail polish, which I am advised makes a good insulator. (I didn’t have any paint-on conformal coating, so used the next best thing.)

I now look forward to testing it in anger on my (allegedly) 90kV supply.

Show and Tell: Geiger-Müller Tubes

Geiger Mueller tubes, with pen for scale.

A collection of Geiger-Müller tubes, used to detect nuclear radiation, with Sharpie marker for scale.

Top to bottom: SI-22G, SBM-20, SI-1G, Anton 1607/BS-212, SI-3BG, SBM-21. Right: SBT-11

Note that, barring the 1607/BS-212, all designations are transliterations from the Cyrillic – these being tubes from the former Soviet Union.

The 1607/BS-212 is also the odd-tube-out in that it operates from a 900V supply; the others all operate at 400V, and can thus all be used with the same counter.

The 1607/BS-212 and SBT-11 both have mica windows, allowing them to detect alpha particles. Alpha particles (actually helium nuclei) are unable to penetrate the walls of the other tubes, so those others are only sensitive to beta particles (high speed electrons) and gamma rays. Mica windowed tubes can be very fragile; luckily Soviet-era engineering makes the SBT-11 relatively robust, but the American 1607/BS-212 needs very careful handling indeed.

For the purposes of monitoring background radiation, the big SI-22G is looking like a winner, being incredibly sensitive. Had I not managed to destroy my test circuit, I might have been able to quantify “incredibly sensitive” a little better – but not until I’ve built a new circuit board. A good test of sensitivity will be the banana test – bananas containing Potassium-40, which emits beta particles.

The SI-1G and SI-3BG most likely won’t end up in equipment. If I were able to detect anything with these mostly hard-gamma tubes, I probably wouldn’t be returning to report on it.

Testing the SBM-20, using a thoriated gas mantle as a check source. This terrible lash-up actually worked, up until the point when I bumped up the supply voltage sufficiently to destroy the circuit. Oops.

Testing GM tube


Show and Tell: Vintage Japanese Milliammeter

Vintage Japanese milliammeterDSC_2449_lzn

It was when I was buying a Kokeshi as a present for my wife that I spotted this lovely old instrument, fitted in its own little suitcase.

I didn’t just buy it because I liked the look of it – it’s a very easy to read, mirror-scaled 0-15mA instrument that I can use in the lab. On the list for calibration, once I have built the required equipment.

I have no idea what the writing on the case says – I just hope it’s not rude, now that I’ve gone and posted it.

Show and Tell: Electrostatic Voltmeter

Electrostatic voltmeter

One of my particular interests in measuring all of the things is measuring instruments, particularly vintage ones.

This electrostatic voltmeter just happened to turn up in the post yesterday, so I thought that a little show-and-tell might be in order.

What’s so great about an electrostatic voltmeter? It all comes down to the act of measuring changing the value being measured. (Very Heisenberg, I know.)

Most of my voltage measurements are taken using a digital multimeter (DMM) which, for most purposes, is just fine. The problem with the DMM, however, is that it has a relatively low input impedance, especially on the higher voltage ranges. Some high voltage power supplies, like those used in Geiger counters, aren’t designed to deliver much current. Try to measure one with a DMM, and the voltage gets dragged down – so the measured voltage is not the normal operating voltage of the unit.

To measure such power supplies, we need an instrument with as high an input impedance as possible. Whilst we could use a DMM connected in series with a high value resistor (>1GΩ) and do some maths, electrostatic voltmeters have an effectively infinite (for a given value of “infinite”) input impedance, making them ideal for the job.

With the instrument pictured, I can measure up to 500V without loading the circuit being measured. Combined with a high resistance voltage divider, I can go higher still. But my super-duper high voltage divider, currently under construction, is the subject of a future post. Can’t have too much fun all at once, can we?

Measure All Of The Things

Until recently, I never really attempted to make a coherent overview of the various projects in which I have been dabbling over the last few years, but the time has come where that needs to change.  My thoughts need better organising than than random scribblings in various Moleskine notebooks – a method that leaves much to be desired when unable to find the notebook in question and certainly not much use when it comes to sharing information with other people. I hope that, by creating this site, there will less fossicking around trying to retrieve information, both for myself, and others.

Recently, trying to summarise many of my interests for someone, I borrowed from the popular “all the things” meme and described my desire to “measure all of the things.” The phrase grew on me, so here we are, on a site dedicated to the discussion of measuring all of the things. It’s probably as well that my phrase included the word ‘of,’ because just imagine trying to register a domain based on the letters ‘MATT.’

Lastly, credit where credit’s due – this WordPress site is doing pretty much what I want it do through the use of plugins from the excellent Joe Dolson. Thank you, Sir.