Why would I write data to a binary data file? It’s all about speed and space.

NSKeyedArchiver in a Nutshell

The NSKeyedArchiver class accepts data objects from your application and stores them on disk. It stores the objects in a file as binary data.

Encoding Diagram
You can open a binary data file in a text editor, but it’s quite hard to make any sense of the contents that way. Binary files are designed to be easily read by the computer, not a person. As a result, binary files take up less space on disk than plain text files and can be loaded by an application very quickly.

Interface Builder, for example, typically stores NIB files in a binary format.

Extracted from this excellent article:

 

Posted in Default Category | Leave a comment

Read and write binary files with Python – Using WS3600’s history.dat file

I did copy this from: Chicoree.fr is the personal site of Sylvain Leroux
This whole thing is available on Github. It’s Very Good. I put it here so I can keep track of it myself. This is “Good Information”

This is the google translated page from:

The Original in French

There are some (argh!) years, I wrote the article read and write a file text with Python . Given the statistical consultation on this page, the thought to myself that it might be time to write its counterpart, namely as read and write a file binary with Python . Even if the Unix tradition is to focus format text , in practice we are often faced with binary data. This is particularly essential when it will process images or sound.Or when it will extract data from a file generated by a proprietary commercial software format. It is precisely in this context that I am writing this article because I am going to work on a historical data file generated by the HeavyWeather associated with a weather station WS3600 software.

Available on GitHub:All sources of this article and the data files that are used as supports are available on GitHub in the project s-leroux / HeavyWeather-WS3600 .

At your convenience you can:

  • Online explore these files;
  • download a ZIP archive of this project;
  • or clone this project if you have a Git client on your machine: git clone git: //github.com/s-leroux/HEAVYWEATHER-WS3600

To begin

What is a file binary ?

To answer Normand, any file that is not a file text file is a binary .

More pragmatically, a file text file is where the data is stored in a form human readable allowing them to change their consultation when using a text editor . This facilitates access to the contents of the file by a user and improves interoperability. In return, this textual representation generally imposes an extra cost of work for the interpretation of the data processing program.

It is therefore opposed to the concept of file binary where the data is stored in a close to that size can directly manipulate the machine – for example promoting the same encoding binary for numbers that one can manipulate the microprocessor. Adopting this representation can improve processing performance and sometimes make the smallest file. Under a less technical aspect, the use of a binary is sometimes also used as a technique of obfuscation by the software vendors to make their products essential to the use of data.

Note:It’s a fair definition considering written in a language relatively low as C. In this case programs, the binary representation of the data in the file is very close to the format of the data in memory when the program manipulates.

For higher-level languages ​​or scripting languages ​​including Python is part, things are put into perspective as you will find that the treatment of a binary file seems more complicated than a text file. It may sound paradoxical, but it is because in reality most of the work on the handling of textual data is taken care of by his or Python libraries, which hides the complexity from the programmer.

Binary or text file format.png

Encoding binary or text  – The same data (here, the wind speed and direction in meteorological data file) can be encoded as binary or as text .

Encoding binary uses a close representation of the machine. For example, by encoding the floating-point numbers in accordance with the standard IEEE 784 , identical to that used by the microprocessor. Different stored data can use different encodings depending on which is most suitable. The operation of binary data requires specialized software.

With encoding text the text representation is decomposed (“human readable”) data characters that are encoded in the file using the same character set as ISO / IEC 8859-1 or ASCII . Data can be reviewed or edited with a text editor or any other software that can read the character set used.

The support file

As I said in the introduction, the file that I use as support is the one generated by a weather station WS3600 type of mark Heavy Weather . Specifically, I will illustrate this article with the test file test_data.datavailable for download in the French section of the manufacturer’s website. This will allow you to redo all the operations using the same data as me:

sh $ mkdir ws3600 
sh $ cd ws3600 
sh $ wget http://www.heavyweather.info/new_french/3600history/test_data.dat

# File generated by the "real" operating system of a 
# WS3600 station named 'history.dat' 
sh $ mv test_data.dat history.dat

Byte Play

I do not know if the definition I gave above was clear enough, but what must be understood is that a file, whatever it is, is a byte on. It is the interpretation of those bytes that will not be the same between a text file and a binary file.

Read a binary file in Python can be done with the same functions as that for a text file. Namely with open , read and close , as you’ll see in the example below where I read a file byte by byte :

# The 'b' to change Farming is for prevent erroneous 
# conversion of end-of-line characters On Some platforms. 
f = open ( "history.dat" , "rb" )
 
try :
     while  True :
	bytes = f. read ( 1 )  # read the next / byte / 
	if bytes == "" :
	     break ; 
	Do stuff with byte # 
	# eg: print as hex 
	print  "% 02X"  %  ord ( byte [ 0 ] ) 
 
except  IOError :
	 # Your error handling here 
	for this example # Nothing 
	pass 
finally :
    f. close ( )
rbThe only difference with reading byte of a text file is to use the modifier b when calling to open :

f = open ("history.dat", "r b ")

This b tells Python that I want to reread the file as it is – without conversion automagically end of line characters to suit the platform on which the program runs.

If you test the display produced by this program with that of the standard command hexdump (1) , you will see that the file has indeed been read :

My program # 
sh $ python single-read.py | head -16 | sed '{N, N, N, s / \ n / / g}'
55 55 55 55 
7B C7 E2 40 
66 66 78 44 
7E 66 66 44 
# With hexdump (1) 
sh $ hexdump v e '1.4 "% 02X" "\ n"' <history.dat | head -4
55 55 55 55 
7B C7 E2 40 
66 66 78 44 
7E 66 66 44 

Okay, but read the file, it is nothing: what is really useful is to interpret the content. Ie to be able to decode the binary data stored there. But this requires knowing how the data are organized and in what format they are encoded. If the binary file format is a format standard , you’re in luck: This information is documented. If this is a format well known , it is likely that you find the information you need on the Internet. If it is an obscure proprietary format , it’ll just make reverse engineering before they can go further …

Decoding File

File Format example

With the support that I use an example, I’ve been pretty lucky. Indeed, although whether it’s a proprietary format and not documented officially , we found on the internet the information necessary for the operation of binary files generated by the WS3600 weather station [1] :

offset (byte) Type C Description Unit
00 double [8] Date Fractional day since December 30, 1899 0:00:00 (GMT)
08 float [4] Atmospheric pressure (abs) millibars
12 float [4] Atmospheric pressure (rel) millibars
16 float [4] Wind speed m / s
20 time consuming [4] Wind direction 0-15 graduated from North (0) clockwise: N, NNE, NE, ENE, …
24 float [4] Wind Speed ​​(gust) m / s
28 float [4] Precipitation (total) millimeters
32 float [4] Precipitation (new) millimeters
36 float [4] Temperature under shelter Celsius
40 float [4] Outside temperature Celsius
44 float [4] Humidity in greenhouses %
48 float [4] External moisture %
52 time consuming (?) [4] ??? always 0

This file format is not going to be too complicated to decode since each record is always 56 bytes. From a more pedagogical view, you will notice that three different encodings are used for numbers:

double
floating point number encoded in 8 bytes
float
floating point number encoded on 4 bytes
time consuming
4-byte integer encoded

Last note, the order of bytes in these numbers follows the convention little-endian – which is easily understood data intended for by the manufacturer to be used on a Windows platform under x86 and x86_64 (this said, I guess the onboard weather station in the microcontroller is also little-endian ).

Read one record

Now that we have a clear idea of the file format, go to its decoding by Python . As usual, there is a standard library for help. Of course, not to decode the particular format, but binary format in general . This module struct . So I can have some modifications to the original program:

import  struct
 
f = open ( "history.dat" , "rb" )
 
try :
     while  True :
        record = f. read ( 56 ) 
        if  len ( record )  ! = 56 :
             break ; 
        # Do stuff with record
 
except  IOError :
         # Your error handling here 
        for this example # Nothing 
        pass 
finally :
    f. close ( )

Note the changes in the code from the previous iteration:

  • I import the module struct – it will help us in the next section, but at least I will not forget;
  • I read the file with package of 56 byte – one record at a time;
  • I interrupt the read loop as soon as I can no longer play a recording complete .

If you want, you can always test the program in a state, but since I do nothing of the record read, no output will be produced. Indeed, before making any treatment must decode each record.

Decode a record

format Type C
equivalent
Encoding
 d double IEEE 754 binary64
 f float IEEE 754 binary32
 l time consuming 32-bit 2’s complement

The Python module struct is specifically designed to decode (and encode) binary data. The key to the operation of this module is to use a screen for describing the structure of a recording in the form of a string of characters Python. In this format, each kind of data corresponds to a letter. The official documentation gives the set of codes for the format. For example, the table below shows against those required to decode a record of my file.

Additionally, these type specifiers, the format used by struct also allows you to specify the first character the endianness . Here we are on data encoded in little-endian which is the character < ( less than ).

Synthesizing all this information with the structure of a record, I come to the following definition of size:

endianness check in
little-endian double float float float time consuming float float float float float float float time consuming
< d f f f l f f f f f f f l

If I tell you that finally provided with this format, you simply create an instance of the class struct.Struct then invoke the method unpack to decode a record, you should understand the changes in the following code:

// ...
 
try :
    s = struct . Struct ( "<dffflfffffffl" )  # Binary data size 
    while  True :
	record = f. read ( 56 ) 
	if  len ( record )  ! = 56 :
	     break ;
 
	print s. unpack ( record )  # Decode ("unpack")
 
// ...
sh $ python struct-read.py | head -3
(38459.854166666664, 993.5999755859375, 1017.5999755859375, 0.20000000298023224, 13, 51.0, 0.0, 0.0, 24.099998474121094, 18.599998474121094, 30.0, 65.0, 0)
(38459.854861111111, 993.5999755859375, 1017.5999755859375, 0.20000000298023224, 13, 51.0, 0.0, 0.0, 24.099998474121094, 18.599998474121094, 30.0, 65.0, 0)
(38459.855555555558, 993.5999755859375, 1017.5999755859375, 3.0999999046325684, 13, 5.8000001907348633, 0.0, 0.0, 24.099998474121094, 18.599998474121094, 30.0, 65.0, 0)

And now: the work of decoding is finished. As you can see, the method unpack returns recording decoded in a tuple . The rest of the program will be the information with Python classic operation of this kind.

To refine

Just to complete the program and do something useful, I’ll add the code necessary to put the date in a more usable form with Python and, say, look for extreme values ​​of temperature in the data file.

InstallationIn Debian you need the package python-tz for the following. If this software is not already on your machine using the package manager of your distribution to install.

sh # apt-get install python-tz

Apart from the import module pytz the modifications are concentrated in the following code fragment:

origin = datetime . datetime ( 1899 , 12 , 30 , 0 , 0 , 0 , tzinfo = pytz. utc ) 
    min = None 
    max = None
 
    s = struct . Struct ( "<dffflfffffffl" ) 
    while  True :
        record = f. read ( 56 ) 
        if  len ( record )  ! = 56 :
             break ;
 
        fields = s. unpack ( record ) 
        ( timestamp, temp ) = ( fields [ i ]  for i in  ( 0 , 8 ) )
 
        # Adjust time 
        timedelta = datetime . timedelta ( timestamp )
        date = origin + timedelta
 
        # Record min and max 
        if  min  is  None  or  min [ 1 ]  > Temp:
             min = ( time, temp ) 
        if  max  is  None  or  max [ 1 ]  < Temp:
             max = ( time, temp )
 
    print  "min" , min 
    print  "max" , max

As you can see, it’s all that Python has more ordinary. And nothing specific to the fact that the data is from a binary file or elsewhere!

sh $ python struct-read.py
min (datetime.datetime (2005, 4, 22, 4, 30, tzinfo = <UTC>) 17.400001525878906)
max (datetime.datetime (2005, 4, 18, 16, 28, tzinfo = <UTC>), 28.5)

Write a binary file

In presenting the module struct I said earlier that it allows not only to decode, but also to encode binary data. This allows consideration not only reread , but also create or edit a binary file.

To illustrate this possibility, suppose that we have noticed a malfunction of the weather station: due to a calibration problem of the corresponding probe, the outdoor temperature is systematically underestimated by 1.4 ° C. With the foundation we have seen, it is possible to write a program to read the data file, a correction, and save the corrected data. The only news reside in the use of the method stdtypes.file.write towrite to a file and struct.Struct.pack to encode a binary structure. These are symmetric in their operation methods read and unpack – it should be no problem if I present you the code directly:

import  struct 
import  datetime
 
# Source file for reading (r) b 
src = open ( "history.dat" , "rb" )
 
# Destination file for writing (w) b 
dst = open ( "adjusted.dat" , "wb" )
 
try :
     Record # struct size 
    s = struct . Struct ( "<dffflfffffffl" )
 
    while  True :
	 # Read a record from source file 
	src = record. read ( 56 ) 
	if  len ( record )  ! = 56 :
	     break ;
 
	# Adjust data 
	fields = s. unpack ( record ) 
	=-adjusted list ( v + 1.4  if i == 9  else v for i, v in  enumerate ( fields ) )
 
	# Encode the record and write it to the file dest 
	= s record. pack ( * -adjusted ) 
	dst. write ( record )
 
except  IOError :
	 # Your error handling here 
	for this example # Nothing 
	pass 
finally :
    src. close ( ) 
    dst. close ( )

Running the program produces no output, but creates a new file adjusted.dat :

sh $ python modify-copy.py  
sh $ ls l * .dat
rw-r - 1 Sylvain Sylvain r-- 1,169,336 4 Jun 11:11 p.m. adjusted.dat
rw-r - 1 Sylvain Sylvain r-- 1,169,336 in February 2005 May history.dat

You will find that the original and the copy is the same size – pointing that all records have been copied. Check all the same that the data have been adjusted to correct by comparing the first two recordings of each of these files:

sh $ hexdump C history.dat -n 112
00000000 55 55 55 55 7b c7 e2 40 66 66 78 44 66 66 44 7 | UUUU {.. @ ffxDff ~ D |
00000010 cd cc 4c third 0d 00 00 00 00 00 4c 42 00 00 00 00 | ..l> ...... LB .... |
00000020 00 00 00 00 41 c0 cc cc cc cc 94 41 00 00 41 f 0 | ....... A ... A ... A |
00000030 00 00 82 42 00 00 00 00 05 b0 5b 5b 7b e2 c7 40 | ... B .... [.. [.. @ {|
00000040 66 66 78 44 66 66 7 44 cd cc 4c third 0d 00 00 00 | ~ ffxDff D..L> .... |
00000050 00 00 4c 42 00 00 00 00 00 00 00 00 41 c0 cc cc | ..LB ........... A |
00000060 cc cc 94 41 00 00 F0 41 00 00 82 42 00 00 00 00 | ... A ... A ... B .... |
00000070
sh $ hexdump C adjusted.dat -n 112
00000000 55 55 55 55 7b c7 e2 40 66 66 78 44 66 66 44 7 | UUUU {.. @ ffxDff ~ D |
00000010 cd cc 4c third 0d 00 00 00 00 00 4c 42 00 00 00 00 | ..l> ...... LB .... |
00000020 00 00 00 00 41 c0 cc cc   ff ff 9f 41 00 00 41 f 0 | ....... A ... A ... A |
00000030 00 00 82 42 00 00 00 00 05 b0 5b 5b 7b e2 c7 40 | ... B .... [.. [.. @ {|
00000040 66 66 78 44 66 66 7 44 cd cc 4c third 0d 00 00 00 | ~ ffxDff D..L> .... |
00000050 00 00 4c 42 00 00 00 00 00 00 00 00 41 c0 cc cc | ..LB ........... A |
00000060   9f ff ff 41 00 00 F0 41 00 00 82 42 00 00 00 00 | ... A ... A ... B .... |
00000070

As can be seen, 4 bytes are different in each recording and correspond to the adjusted temperature of 1.4 ° C. If you do not believe me, I’ll let you play with unpack to convince you.

Update a binary file

Next step of our journey, having created a new file from an original left intact, see how to change a file locally . Ie by directly changing the data in the file.

Again, very little difference with the previous version. At most, opening the file mode r + b (read and modified in binary mode). And the use of primitive seek to go back into the file after reading a record to crush it with his updated version:

import  struct 
import  datetime 
import  os
 
# Source file to read year modify (r) b 
f = open ( "history.dat" , "r + b" )
 
try :
     Record # struct size 
    s = struct . Struct ( "<dffflfffffffl" )
 
    while  True :
	 # Read a record from the file 
	record = f. read ( 56 ) 
	if  len ( record )  ! = 56 :
	     break ;
 
	# Adjust data 
	fields = s. unpack ( record ) 
	=-adjusted list ( v + 1.4  if i == 9  else v for i, v in  enumerate ( fields ) )
 
	# Encode the record 
	record = s. pack ( * -adjusted )
 
	# 'Rewind' 'the file back one record 
	. f seek ( - 56 , bone . SEEK_CUR ) ;
 
	# Overwrite the record with the modified version of 
	f. write ( record )
 
except  IOError :
	 # Your error handling here 
	for this example # Nothing 
	pass 
finally :
    f. close ( )

Most sharpened have noticed that at least in theory my program is far from efficient. Indeed, I read and I write each record in the entire file when in reality I have to change 4 bytes per record. In practice, doing so makes the code more readable and maintainable than calculating offsets complicated in the file. Moreover, between the power of a modern machine and the various levels of buffer that come into play, I’m not sure it’s that easy to get a significant gain. So, for that make a choice, so take that for clarity: to quote Donald Knuth : “Premature optimization is the root of all evil.” Especially as you will see in the next section that there is an alternative technique to consider a more effective treatment for this problem. But before we be interested, check nonetheless that this version works as expected:

# Preserving the original data file will be modified as 
sh $ cp history.dat original.dat 
sh $ chmod w original.dat 
# Running the program 
sh $ python modify-in-place.py
# Check the outcome 
sh $ diff history.dat original.dat && echo identiques
Binary files and history.dat original.dat Differ
sh $ diff history.dat adjusted.dat && echo identiques
identiques

The modification on the spot has produced the same file as the change with a copy of the previous section.

Memory Map

Previous techniques used for input / output to the former based on the management by the programmer cycles of reading and writing required to transfer the data block from disk to memory and vice versa.

But modern operating systems offer a more efficient alternative in the form of mapping a file into memory ( Memory-mapped file ). The idea here is to match an address space to a file on disk. The loading and unloading of actual data between the disk and the memory being vested in virtual memory manager, which makes these operations completely transparent to the programmer.

Sounds technical air – and frankly it is a little. But this complexity is supported by the operating system. From the perspective of the programmer, the mapping of a file into memory are usually limited to a function call mmap . After this call, not only everything happens as if the file had been fully loaded into RAM, but also the changes made ​​in the address space mapped are reflected on disk. Thus, in the following code, you will see that I accesses the file mapped in exactly the same manner as if it were a painting in memory. Without worrying about when or how to access data on the disk:

import  struct 
import  datetime 
import  mmap
 
# Source file to read year modify (r) b 
f = open ( "history.dat" , "r + b" )
 
try :
     Record # struct size 
    s = struct . Struct ( "<dffflfffffffl" )
 
    # The Whole Memory map file (size = 0) 
    map = mmap . mmap ( f. fileno ( ) , 0 )
 
    # Walk record by record throught the address space 
    for idx in  xrange ( 0 , map . size ( ) , 56 ) :
	 # Unpack the current record and adjust it 
	fields = s. unpack ( map [ idx: idx + 56 ] ) 
	-adjusted = list ( v + 1.4  if i == 9  else v for i, v in  enumerate ( fields ) )
 
	# Encode the record 
	map [ idx: idx + 56 ] = s. pack ( * -adjusted )
 
except  IOError :
	 # Your error handling here 
	for this example # Nothing 
	pass 
finally :
     map . close ( )   # unmap 
    f. close ( )     # close the file

You have seen, the main difference with the previous versions is the fact that there is more explicit appeal to primitive read / write read and write . Reading and editing of data happens transparently when accessing the variable map :

# Play implicit memory data 
fields = s.unpack ( map [idx: idx + 56] )
# Implied Writing data to disk
map [idx: idx + 56] = s.pack (*-adjusted)
Trap:As so often when it comes to manipulating files, the data is not guaranteed written to disk when the file is closed or when an explicit call to the method mmap.flush .

mmap is actually more effective?At first glance, reducing the number of intermediate buffers – and therefore the number of copies of the data – mmap improves file access performance. But in practice, the actual gain may be negligible or become a loss depending on how you use the file.

The mapping memory is particularly effective when you access repeatedly read write certain portions of the file. Conversely, it is a solution in principle ineffective when sequential access to a file (as here). Indeed, as the expanded memory manager can not guess that you will not need the pages loaded, it tends to keep in mind all the content already read the file. Increasing the RAM consumption, which in turn can lead to increased page faults ( page fault ) and therefore degrade the overall system performance . This description, however, is catastrophic in perspective in the case of a small file with a full load will have a marginal impact on memory consumption. Here, small means compared to the available RAM. In the case that interests me, I do not think loading a file from a few megabytes on a machine where the RAM is measured in gigabytes is very penalizing. In the end, you see that the choice of using mmap or not can be complicated. In practice, unless special requirements, I recommend the technique that produces the easiest code to understand and maintain!

The last word

This exploration of techniques for manipulating binary files in Python coming to an end. If we ended on a slightly more advanced concept of memory mapping , essentially it is about combining conventional techniques / O on files with the features pack and unpack the module struct . If I presented this option is that it stuck well with the file I used as support. But there are others: so if my file had contained homogeneous data (all the same), the module array would probably have been a better choice. If necessary, I think with the explanations given here and the official documentation, you should now be able to understand the operation yourself.

Resources

Posted in Weather | Leave a comment

On Windows? Try ATMEL as another option?

The Atmel Studio works hand in hand with the Arduino IDE. I have just compiled the exact same code as on the Mac, on my Windows 8.1 laptop, uploaded it to the Arduino Uno board – and it works straight out of the box. I can monitor it by using Coolterm in Receive mode.
Advantages? Atmel Studio uses Libraries that are unavailable to the Arduino, like the AVR libraries, ie. avr/io.h and avr/interrupt.h, and avr/pgmspace.h

Probably other stuff too.
I don’t know why the sketch compiles – and works !!!! – as it is on the Arduino IDE, even with those libraries missing? on the OSX-Mac, but it does. It also compiles and works as it is, on the Windows, AtmelStudio setup. Interesting.

ATMEL Logo

Atmel Studio 6.2: Released for Production

Atmel Studio 6 – The Studio to Design All Embedded Systems
Atmel Studio 6.0 Download
Installation Instructions
1.1 System requirements
1.1.1 Supported operating systems
● Windows XP (x86) with Service Pack 3 – all editions except Starter Edition
● Windows Vista (x86 & x64) with Service Pack 1 – all editions except Starter Edition
● Windows 7 (x86 and x64)
● Windows 8 / 8.1 (x86 and x64)
● Windows Server 2003 (x86 & x64) with Service Pack 2
● Windows Server 2003 R2 (x86 and x64)
● Windows Server 2008 (x86 and x64) with Service Pack 2
● Windows Server 2008 R2 (x64
Posted in Weather | Leave a comment

Works for Sparkfun Weather Shield… brilliant. Windows only I think

Get Weather Station Data Logger at SourceForge.net. Fast, secure and Free Open Source software downloads

http://wmrx00.sourceforge.net/

Weather Station Data Logger

for WMR88, WMR100, WMR200, RMS300 and Radio Shack 63-256 Weather Stations

Towering Cumulus

Download Now

News — November 9, 2013: Version 4.4.5 released.

Posted in Weather | Leave a comment

Calculating Altitude, using BOSCH formula, and Constants for Sealevel.

This is from the BOSCH Data Sheet.

AltitudeFormula

And this is the code implementation, using the standard measurements for SeaLevel, Pressure and Temperature as defined by NIS
Atmospheric properties[edit]
At SSL some atmospheric properties are:[1]

Pressure, P = 101.325 kPa = 2116 lbf/ft2
Density, \rho = 1.225 kg/m3 = 0.002377 slug/ft3
Temperature, T = 288.16 K = 518.69 °R = 15.01 C
=============
So you can use 20C or 15.01 … I doubt it makes much difference. Just don’t take your house into the air to any great height…

The WeatherShield output shows me an Altitude of 18.08, which oddly enough is about right. Surprises even me!

80,27,27,16.8,49,-,WSW,8.0,W  ,3.8,1011.70,18.08,22.5
...........................................^^^^^
  /* The formula given by BMP183 Bosh mfg, is
    altitude = 44330*(1-(p/p0)^(1/5.255)) which if used with the CONSTANTS, should determine a true altitude.
  */
    
  //Implementation in the weather shield code. The variable is Pzero by the way. P0, not uppercase PO
  const float P0 = 1013.25; // seaLevel,
  		ut = bmp180_get_value(BMP180_TEMPERATURE, 5);
		t = (bmp180_get_temperature(ut));
		up = bmp180_get_value(BMP180_PRESSURE_3, 26);
		p = bmp180_get_pressure(up);                 

  //float T = t/10;
  float T = 15.01; // standard sea level temperature
  float P = p/100; // atmospheric pressure in hpa
  float h, altitude; //height in meters
  
  //h = (((float)pow((P0/P), 0.190223F) - 1.0F) * (T + 273.15F)) / 0.0065F;
  // altitude = h;
  altitude = 44330.0*(1.0-((pow((P/P0), (1.0/5.255)))));  //BOSCH code
  	
Posted in Weather | Leave a comment

Images of processes

2014-08-19 13.07.20

Screenshot 2014-08-19 13.06.07

2014-08-19 13.13.06

Posted in Weather | Leave a comment

Some thoughts and finding on Altitude as Weather Enthusiasts might find interesting.

Mostly from Wiki pages

Calculates the altitude (in meters) from the specified atmospheric
pressure (in hPa), sea-level pressure (in hPa), and temperature (in �C)

@param seaLevel Sea-level pressure in hPa
@param atmospheric Atmospheric pressure in hPa
@param temp Temperature in degrees Celsius

National Institute of Standards and Technology (NIST), uses a temperature of 20 °C (293.15 K, 68 °F)
and an absolute pressure of 101.325 kPa (14.696 psi, 1 atm).

Standard sea level (SSL) (also known as sea level standard (SLS)) defines a set of conditions for
physical calculations. The term “standard sea level” is used to indicate that values of properties
are to be taken to be the same as those standard at sea level, and is done to define values for use in general calculations.

Atmospheric properties[edit]
At SSL some atmospheric properties are:[1]

Pressure, P = 101.325 kPa = 2116 lbf/ft2
Density, \rho = 1.225 kg/m3 = 0.002377 slug/ft3
Temperature, T = 288.16 K = 518.69 °R = 15.01 C

Pressure altimeter
Digital barometric pressure sensor for altitude measurement in consumer electronic applications
Altitude can be determined based on the measurement of atmospheric pressure. The greater the altitude the lower the pressure.
When a barometer is supplied with a nonlinear calibration so as to indicate altitude, the instrument is called a pressure
altimeter or barometric altimeter. A pressure altimeter is the altimeter found in most aircraft, and skydivers use wrist-mounted
versions for similar purposes. Hikers and mountain climbers use wrist-mounted or hand-held altimeters,
in addition to other navigational tools such as a map, magnetic compass, or GPS receiver.

The calibration of an altimeter follows the equation

z=cT log(P0/P) where c is a constant, T is the absolute temperature, P is the pressure at altitude z, and Po is the pressure at sea level.
The constant c depends on the acceleration of gravity and the molar mass of the air.
However, one must be aware that this type of altimeter relies on “density altitude” and its readings can vary by hundreds of feet owing to a
sudden change in air pressure, such as from a cold front, without any actual change in altitude.

Or to put it another way. If you have a stationary – ie, in the house – weather station, and you suddenly notice your altitude is dramatically increasing, look out the window
there is probably a hurricane heading toward you…. low pressure = higher altitude. Otherwise, your recorded altitude will only vary according to local air pressure
variations, and generally, won’t change much

*/
/**************************************************************************/
/* Hyposometric formula: */
/* */
/* ((P0/P)^(1/5.257) - 1) * (T + 273.15) */
/* h = ------------------------------------- */
/* 0.0065 */
/* */
/* where: h = height (in meters) */
/* P0 = sea-level pressure (in hPa) */
/* P = atmospheric pressure (in hPa) */
/* T = temperature (in �C) */



  const float P0 = 1013.25; // seaLevel,
  		ut = bmp180_get_value(BMP180_TEMPERATURE, 5);
		t = (bmp180_get_temperature(ut));
		up = bmp180_get_value(BMP180_PRESSURE_3, 26);
		p = bmp180_get_pressure(up);                 

  //float T = t/10; //current temperature. Test both, varies very little
  float T = 15.01; //Temp standard for sealevel
  float P = p/100; // atmospheric pressure in hpa
  float h, altitude; //height in meters
  
  h = (((float)pow((P0/P), 0.190223F) - 1.0F) * (T + 273.15F)) / 0.0065F;
   altitude = h;

This is producing an Altitude that currently varies by about 5 meters at my place over a few days.

Posted in Weather | Leave a comment

as3935-lightning-sensor-board

https://www.tindie.com/products/TAUTIC/as3935-lightning-sensor-board/

Posted in Default Category | Leave a comment

Converting and creating LaCrosse history.dat files

 

http://www.lissproductions.org/wuhu_manual/the-wuhu-software-manual/wuhu-history-data-file-format/

WUHU History data file format

There are two historical files that WUHU maintains.

“WUHU History.dat” is for internal use by WUHU.

It’s basically an image of the memory in case of power failure. It contains every reading your station has made in the past X hours where X is set in your general settings. You can specify up to 1 week of data that can be kept in memory. This setting effects how much data you can chart and display statistics for.

“History.dat”

For use with other programs, always use the history.dat file output by WUHU.

Here is the format of the history.dat file that is output by WUHU. This matches the file format output by the Heavyweather Pro application. The HW Pro application supports WS-36XX stations. We use the HW Pro application to graph the history files output by WUHU.

This format was chosen because it contains more fields than the old HW Beta 2.0 format.

New records are added to the end of the file, there is no header or trailing record.

The [X] field denotes the number of bytes in the field.

  1. 00 Double [8] Timestamp days from 12/30/1899 00:00:00 (GMT)
  2. 08 Float [4] Abs Pressure hectopascals (millibars)
  3. 12 Float [4] Relative Pressure hectopascals (millibars)
  4. 16 Float [4] Wind Speed meters/second
  5. 20 ULong [4] Wind Direction see below
  6. 24 Float [4] Wind Gust meters/second
  7. 28 Float [4] Total Rainfall millimeters
  8. 32 Float [4] New Rainfall millimeters
  9. 36 Float [4] Indoor Temp celsius
  10. 40 Float [4] Outdoor Temp celsius
  11. 44 Float [4] Indoor Humidity %
  12. 48 Float [4] Outdoor Humidity %
  13. 52 ULong [4] unknown – (Value is always 0)

Wind Direction

Wind direction is encoded as an integer between 0 and 15. To get the wind direction in degrees, multiply the value by 22.5. To get compass directions (moving clockwise) 0 is North, 1 is North-Northeast, 2 is Northeast, etc…

. 0 N
. 1 NNE
. 2 NE
. 3 ENE
. 4 E
. 5 ESE
. 6 SE
. 7 SSE
. 8 S
. 9 SSW
. 10 SW
. 11 WSW
. 12 W
. 13 WNW
. 14 NW
. 15 NNW

C data structure that mimics this format:

struct WeatherRecord3600 {
double fTimeStamp;
float fOutdoorAbsolutePressure;
float fOutdoorRelativePressure;
float fOutdoorWindSpeed;
unsigned long ulOutdoorWindDirection;
float fOutdoorWindGust;
float fOutdoorTotalRainfall;
float fOutdoorNewRainfall;
float fIndoorTemperature;
float fOutdoorTemperature;
float fIndoorHumidity;
float fOutdoorHumidity;
unsigned long ulUnknown;
};

The fTimeStamp is an odd field. It represents the days since a epoch time (12/30/1899 00:00:00) in UTC local time. The fractional part is used to obtain hours, minutes, and seconds. There are a few functions that are needed to convert back and forth between that timestamp format and the normal local time format, they are given below.

To convert from a normal time_t to this format you would use:

time_t t;
WeatherRecord3600 wr3600;

wr3600.fTimeStamp=Convert_time_t_To_DateTime2(ConvertLocalToUTCLocal(t));

To convert from this odd timestamp to a time_t:

t=ConvertUTCLocalToLocal(Convert_DateTime2_To_time_t(wr3600.fTimeStamp));

-----------------------

double Convert_time_t_To_DateTime2(time_t vtime_t)

{
double fresult;
// 25569 days from Saturday, December 30, 1899 to Sunday, January 1, 1970 (where time_t seconds counts from)
double fDeltaDays = 25569 + (vtime_t / (60 * 60 * 24));
double fFractionDays = (vtime_t % (60 * 60 * 24)) / (double)(60 * 60 * 24);
fresult = fDeltaDays+fFractionDays;
return (fresult);
}

//----

time_t Convert_DateTime2_To_time_t(double fTime)

{
return (time_t)((fTime - 25569.0) * 86400.0);
}

//----

time_t ConvertUTCLocalToLocal(time_t t)

{
// Aquire a local time and a gmt time.
struct tm tm_localtime;
memcpy(&tm_localtime,localtime(&t),sizeof(tm_localtime));

// Aquire UTC time.
struct tm tm_gmtime;
memcpy(&tm_gmtime,gmtime(&t),sizeof(tm_gmtime));

// Convert them back to time_t types.
time_t t1=mktime(&tm_localtime);
time_t t2=mktime(&tm_gmtime);

// Subtract from local time the difference between local time and gmt time.
t1 += (t2-t1);

// If daylight saving time in effect, subtract an hour.
if (tm_localtime.tm_isdst)
t1 -= 60*60;

return(t1);
}

//----

time_t ConvertLocalToUTCLocal(time_t t)

{
// Aquire a local time and a gmt time.
struct tm tm_localtime;
memcpy(&tm_localtime,localtime(&t),sizeof(tm_localtime));

// Aquire UTC time.
struct tm tm_gmtime;
memcpy(&tm_gmtime,gmtime(&t),sizeof(tm_gmtime));

// Convert them back to time_t types.
time_t t1=mktime(&tm_localtime);
time_t t2=mktime(&tm_gmtime);

// Subtract from local time the difference between local time and gmt time.
t1 -= (t2-t1);

// If daylight saving time in effect, add an hour.
if (tm_localtime.tm_isdst)
t1 += 60*60;

return(t1);
}

 

Posted in Weather | Leave a comment

A collection of interesting stuff

 

http://www.lissproductions.org/wuhu_manual/2011/11/29/diy-lightning-detectors/

and just in case it gets lost:

DIY lightning Detectors

Hi Everyone!

I began writing this email to ask if anyone had come across a DIY lightning Detector that could be mapped on a PC.  I ended up with a list of great links that I thought I would share with you!

I did find Site Mate.  It’s a FREE program to look at personal Lightning Radar sites (NO hardware required)  http://www.lrsatx.com/sitemate_page.htm

Simple lightning detectors – (clicks or flashes a light for each storm strike)

* Schematics for Lightning Detectors
http://www.techlib.com/electronics/lightning.html#Egor
This is the same one that Dimitris just posted.  Here’s his link:
http://users.otenet.gr/~meteo/project_portable-lightning-detector.html

Note that in the “reader’s versions” of the first link, one of the readers used this circuit to send a pulse to the electronic trigger of his digital camera.  This allows him to release the shutter at precisely the same time that a strike hits.  This is a GREAT idea, as anyone who’s sat in the rain with a “bulb” exposure will attest!  :-D

* Hobby Boards Lightning Detector ($32.50 assembled) Also available as a kit.
http://www.hobby-boards.com/catalog/product_info.php?cPath=22&products_id=1550

DIY Lightning Radar – (shows location of strikes on map using PC)

Personally, I was looking for DIY hardware that could produce results similar to Strikestar :

“StrikeStar allows multiple, standalone lightning detectors to form a real-time lightning locator network with much better positional accuracy.”

Strikestar is  “exclusively designed for the NexStorm software and Boltek hardware “.  Needless to say, the Boltek hardware ($599) and software ($135+)  package is very expensive.

I found two different DIY systems using two different approaches:

1. Lightning radar MDF (= magnetic-direction-finding system) by Frank Kooiman
2. TOA (= time-of-arrival system) by Egon Wanke

Gerald Iihninger’s lightning detection page gives a good overview of each system.
http://members.inode.at/576265/lr.htm

1. Lightning radar MDF (= magnetic-direction-finding system) by Frank Kooiman

What is Lightning Radar?

Lightning detector system with:
* 2 crossed loop antenna’s tuned to 10 kHz
* 2 simple opAmp amplifiers with gain of 100 x
* a sound card of a 1 GHz PC
* a free program that detects the direction to the source
of the lightning strike with an accuracy of 1 degree.
This program can be used at different sites to calculate
the location of the lightning strike using a trangulation method.

This system was developed as a hobby alternative to  the existing commercial Boltek lightning detector. The  advantages of the lightning radar are the low cost (€40 and up) compared to the Boltek (€350 to €600 depending on the version), the extreme sensitivity of the system, and the possibility of joining the group system  via the internet. Where Boltek detectors can detect lightning up to a range of 500km, the LR (lightning radar) has a range of 2000 to 3000km over land and several thousand km over water (e.g. lightning in Florida, south America).

One disadvantage of the LR is that it is not a plug-and-play system and therefore requires some knowledge of electronics and familiarity with a soldering iron. In practice, this is not really a disadvantage since it means that you learn a lot more about the science of detecting lightning.

Links:

Wouldn’t ya know it, but at the end of my research, I found Dimitris’ site!  I guess he’s the resident expert here!  :-D

* Dimitris site!
o http://users.otenet.gr/~meteo/project_lightning-radars.html
* Amateur Lightning Detector and Radar by Frank kooiman
http://members.home.nl/fkooiman/lightning/index.htm
* Partner Ground Station “Lightning Radar Project”
o http://users.edpnet.be/DanielV37/Detecteur3/
* LightningRadar.net
o http://www.lightningradar.net
* Links for LR Stations
o http://www.lrsatx.com/lightning_radar_1.htm
* San Antonio, US site w/description of Site Mate (and link to it too)
o http://www.lrsatx.com
* Site Mate – FREE program to look at personal Lightning Radar sites
o http://www.lrsatx.com/sitemate_page.htm

2. TOA (= time-of-arrival system) by Egon Wanke
http://www.blitzortung.org/Webpages/index.php?mode=3&map=0&lang=en

This system uses a pre-amp circuit board, evaluation board, VLF antenna (ferrite rods or loop ant above), and GPS with one-pulse-per-second (1PPS) output & serial interface

3.  Commercial Systems

* Stormwise – systems and components (ferrite rods, Specialty Directional Antennas)
o http://www.stormwise.com
* Boktek – Stormtracker, LD-250, etc…
o http://www.boltek.com
* Strikestar – software for Boltek systems
o http://www.strikestarus.com

I hope this helps (and inspires) someone else!  Thanks to Dimitris for bringing up the subject!

Jon G.

===========================================

 

http://www.lissproductions.org/wuhu_manual/2011/11/26/lacrosse-specific-information-hardware/

 

LaCrosse Specific Information – hardware

LaCrosse Weather station manuals

Here is a list of PDF‘s for your LaCrosse Weather station(s):

http://site.ambientweatherstore.com/Manuals/

LaCrosse 23xx Cable Pinout

Pinout for the RS-232 cable for the WS-2310/15/17

On RJ-11 Side (clip down, telephone style connector)
From left to right looking down on the connector
Color: Blue Brown Green White

On the 9 Pin D-Shell (PC Interface)
Signal: RTS RXD TXD DTR
Pin...: 7 2 3 4

So Blue -> Pin 7, Brown -> Pin 2 etc.

Note that there is NO connection
to the RS-232 ground (pin 5)!!!

=================================

LaCrosse WS-2810 Full Startup/Reset Procedure

  1. If the battery in the wind sensor has not charged in full sun (facing due south) for at least three days, do this first.
  2. For this procedure, chances of success are better when the distances between the display unit and other sensors relative to the temp/humidity sensor are within 50 feet. Line-of-sight orientation and fewest possible intervening obstructions are desirable. Further separation can be done after start-up.
  3. After solar battery is charged, go to the display unit and follow the following steps as, and in the order, listed.
  4. Press and hold the Set button. Release when the display ‘blanks out’.
  5. Repeatedly press and release Set button until the “rES off” message appears at bottom of screen.
  6. While in the “rES Off” mode, press and release the Up Arrow to change the message to “rES On”. Then press the Set button (once).
  7. A counter at the bottom of the page will now appear showing the value “127” and then it will begin counting down to “0”. When zero is reached, the display will show “rES dOnE”.
  8. At this point, remove the batteries from the display.
  9. While batteries are still out, press any key on the display at least 20 times. (This helps to drain any residual charges in the circuits)
  10. Now remove the batteries from the rain sensor and the temp/humidity sensor.
  11. Leave everything as-is for at least 10 minutes.
  12. Now go to the wind sensor and press and release the reset button recessed in the little hole on the bottom of the wind sensor. Use the provided plastic ‘rod’ or a straightened paper clip. The switch is approximately 5/8” straight inside the case. The actuation requires very little pressure. If you do it carefully you can feel (but probably not hear) a soft switch movement.
  13. Now replace the batteries in the rain sensor and temp/humidity sensor, followed by replacement of batteries in the display unit.
  14. Leave everything as-is and display untouched for at least 10 minutes.
  15. Everything should now be linked and the display should show connectivity and activity of all the sensors.
  16. If something is still not functioning, you should call LaCrosse technical support.

 

=================

http://www.lissproductions.org/wuhu_manual/2011/11/26/lacrosse-23xx-wind-sensor-modification/

 

 

Posted in Weather | Leave a comment