When I first started making AVR modules, there was no Raspberry Pi or ESP8266. Arduino was introduced in 2005, the same year I started with the AVR microcontroller. But I didn’t know of it until many years later.

There weren’t any Wi-Fi enabled microcontrollers at that time. Both Ethernet and Bluetooth interfaces were more expensive, and harder to interface. So I ended up using the good old serial port.

I made my first module with a serial interface in 2007. RS-232 was cheap, and easy to implement.

With RS-232 I got a way of getting data between the microcontrollers and the computer, but I still needed a defined syntax. So I set out to make one, and called it SIOS — Serial Input/Output System.

Table of contents

Motivation

My main project at the time, The Rack box, was getting big. Everything was controlled by microcontroller modules, acting on input signals. I wanted to bring all I/Os into the computer, which would allow me to create more advanced automations — quicker and easier.

The rack box project — April 2009

The status panel for the rack box, consisting of LEDs and switches, was also getting full. Any changes or additions were time consuming, and difficult. Instead I wanted a software status panel that I could easily update and expand 😃

Rack box status panel — November 2007

Syntax

I made some big changes to the syntax in 2013 — this became the 2nd revision. With rev. 2 the formatting and encoding was changes, but both revisions had the same basic parts:

  • Module address
  • Request type
  • Request ID
  • Request value
  • Checksum

Revision 1

  • : was used as a delimiter
  • numeric values in base-10
  • addressing started on 001 (hard coded)
  • baud rate was 9600
001:o:3:1:132

The command above tells module 001 to set its output o number 3 to true 1.

Revision 2

  • multiple delimiters used, to make it easier to identify different segments of the command
    • , for identifier (address, type and ID)
    • : for value
    • # for checksum
  • base-16 (hex) used for numeric values
  • addressing made programmable; 32 to 125 (0x20 to 0x7D)
    • with 126 (0x7E) used for broadcast
  • digital IOs addressed as byte, not individually
  • module status and status bit was introduced
  • baud rate bumped to 38400
20,o,0:4#84

The command above tells module 0x20 (32) to set its digital output byte to 4, which means turn on output 3 (you know; 1 2 4 8…)

Module address

There really was no need for a module address; as RS-232 can only be between two devices. But with addressing it was possible to have multiple modules connected to a single host, without the client knowing what serial port they were connected to.

My plan was to switch to RS-485, which can be multipoint and would require addressing. I only made a single RS-485 module, before switching to Arduino, Raspberry Pi and ESP8266.

Request types

  • Output: o
  • Input: i
  • Status: s
  • Unit setup: u

Request ID

For revision 1 the base-10 request ID was simply the digital IO pin. But for revision 2 this was changed to byte for digital IOs:

  • 0: Digital input or output byte
  • 1-15 (0x1-0xF): Analog input or PWM output

Module status

Module status and status bit was introduced with revision 2, and allowed the module to inform the host of its capabilities.

#Adr Status bit Status Info Type
0 STAT Status 8-bit Byte
1 SERIAL Serial number #Single
2 NAME Unit name String
3 VERFIRM Firmware version String
4 COMPDATE Compiled date String
5 VERBOOT Bootloader ver. #Single
6 VERPROT Protocol version #Single
7 DIO Digital in, out #Double
8 AI Analog in, bit #Double
9 AO Analog out, bit #Double

Status bits

If any module status bit was set, meaning the value was above 0, the status LED on the module was red instead of green.

  • 0: BootFlag
    • set to true on startup, could be manually reset
  • 2: DefAddressFlag
    • set to true on startup if module was using the default address, could be manually reset
  • 4: ManFailFlag
    • default false, manually toggled

Module setup

Also introduced with revision 2. A software option to reboot the module was useful when programming over the serial port. The modules would first be rebooted, then put in programming mode.

#Adr Param Setup Info Type
0 RESET Reboot module Bool
1 ADR Module address ¹ Byte
  1. reboot required if changed

Modules

In total I made six modules. Two IO modules were rebuilt to be stand-alone sensors.

Serial ports

Initially I used a serial expansion card from Comtrol with an external interface. One PCI slot gave me 8 serial ports. I made adapters to turn the DB-25 female into RS-232 DB-9 male.

This worked fine as long as all the modules were in the same place.

Comtrol RocketPort serial interface

When I got a house and started spreading the modules out, I instead used a couple of serial servers and some USB to serial adapters.

Two serial servers (COM to LAN adapters)

The serial servers made a single serial port available on the network through a TCP port. They were quite expensive, and made the whole setup bulky.

USB to serial adapter

Firmware

I wrote all my AVR firmware in BASCOM-AVR. I programmed all modules with a bootloader, which allowed me to update the firmware using the serial port. So much easier than opening the modules and taking the microcontroller out.

The firmware for the different modules is available in git repositories on the respective project pages.

Software

Python

This is a simple, proof-of-concept Python script to send and receive data from a serial module. It sends commands to the broadcast address (126), splits, decodes and verifies the received data.

import serial #pyserial
import re, sys
import functools, operator

def hex2dec(s):
  try:
    return int(s, 16)
  except (ValueError, TypeError) as ex:
    print('Type/Value: %s', str(ex))

ser = serial.Serial('/dev/ttyUSB0', 38400, timeout=1)

adr = 126 # broadcast
s = str(sys.argv[1])

send_str = chr(adr)+','+s+'\r\n'

ser.write(bytes(send_str, 'utf-8'))
response = ser.readline()
line = response.decode("utf-8")

if not line:
    raise ValueError("Empty response")

print(line.rstrip())
pos_adr = line.find(':')
pos_crc = line.find('#')
str_adr = line[pos_adr-6:pos_adr]
str_val = line[pos_adr+1:pos_crc]
str_crc = line[pos_crc+1:len(line)-1] #-1

checksum = functools.reduce(operator.add, map(ord, str_adr + ":" + str_val))
while checksum > 255:
  checksum -= 256

if int(str_crc) == checksum:
    print("Checksum OK")

if re.match('u,1',s):
  print(hex2dec(str_val), str_val)
elif re.match('s,[1-6]',s):
  print(str_val)
elif re.match('s,[7-9]',s):
  print(hex2dec(str_val[0:2]), hex2dec(str_val[2:4]))
else:
  print(hex2dec(str_val))

In the example below, we send the command o,0,F to the module. Serial port and address is set in the Python script.

$ python3 sios.py o,0,F
20,o,0:000F#105
Checksum OK
15

We receive 20,o,0:000F#105, which means:

  • from module 0x20 (32)
  • type is o (output)
  • ID is 0x0 (digital output byte)
  • value is 0x000F
  • checksum is 105

The checksum is verified to match the received string, and the value converted to base-10 is 15. We have turned on digital output 1, 2, 3 and 4.

Demonstration

What happens here:

  • Resetting status bit 1 and 2
    • Status LED turns green when all status bits are off
  • Turn on digital outputs 1 to 4
  • Turn off digital outputs
  • Set PWM output 1 value to 16, 80, 255 and 0
  • Read digital inputs, while activating toggle switch
  • Read analog inputs 1 and 2
  • Toggle status bit 3
  • Read status data values 1 to 9

Serial server

Screenshot of Serial Server

Serial server was written in Visual Basic, not the best language, but it was what I knew at the time. It handled communication with all the serial modules.

  • acted as a gateway between the serial modules and a connected serial client
  • forwarded single TCP commands to the correct serial module
  • converted received analog sensor values to physical units
  • handled timers and triggers
  • received a heartbeat signal from the serial modules every 20 seconds
  • cached the last received states for all IOs for all modules, this was forwarded to the serial client on login

Configuration screenshots

Serial server analog sensor configuration
Serial server timer configuration
Serial server trigger configuration

Bad programming

The application worked, but was badly written. All code related to the serial ports was duplicated for each port, making it a lot of work to add support for additional serial ports.

When there was a lot of traffic to multiple modules; the serial server had problems keeping up, and occasionally lost some strings.

It started small, and grow out of proportions. It remained the weakest link in the system for its entire lifetime.

TCP commands

Serial server could receive commands on a TCP socket, the socket was automatically terminated after the command was forwarded.

I made a Windows application called SerialTCPcommand.exe:

 SerialTCPcommand.exe password;002:o:08:1

In Linux netcat could be used:

 echo "password;002:o:08:1" | netcat ip-address port

Module config files

Each serial module had a config file, e.g. 001.ini, which informed the serial server of the capabilities of the module:

  • Baud rate, data bit, start-char
  • Module name
  • Number of inputs
  • Number of outputs
  • Can receive requests (0/1)
The 2nd syntax revision eliminated the need for these files, as the modules could provide this information directly. But the serial server got scrapped before rev. 2.

Serial server logs

Serial server wrote three kinds of logs:

Historical
Every action stored in one file. Periodically parsed to a MySQL database, and used for historical data.
Value
The converted value of all analog sensors was written in individual files. Used for sharing data with other applications, like mrtg and Serial Client 4.x.
State
Current state of all serial modules and IO ports was written in individual files. Used for sharing data with Serial Client 4.x.

Serial client

The client communicated with the serial modules through the server — like a user interface. It had very limited logic, and relied on the serial server for converted analog sensor values, triggers, etc.

The server and client shared data using a TCP connection and log files.

Version 1

Screenshot of Serial Client v.1

Version 1 of the client was written in Visual Basic, and was terrible. It had a number of major bugs and was unable to handle any latency in the connection to the server, causing it to constantly loose information.

Version 2

Screenshot of Serial Client v.2

Version 2 was written in Flash, which was really hot at the time, and could be run in a browser — which was a major advantage. Flash also made it possible to make a decent UI, and handled latency in the server connection well.

The CCTV photo was captured by a TV input card, which saved it to the web server every 3 seconds.

I even think I experimented with animations and text-to-voice 😛

Version 3

Screenshot of Serial Client v.3

Version 3 was written in ActionScript (still Flash). Basically version 2 with an updated UI and added controls.

Version 4

Screenshot of Serial Client v.4

Version 4 was written in PHP and HTML, with CSS styling. Native web, and leaving that Flash crap behind. Unlike the previous versions; it didn’t need a TCP connection to the serial server, only HTTP(s) to the web server. This meant that it also worked behind corporate firewalls 😃

At this point the limitations of the serial server was really starting to show… It saved state and value files on a Samba share, that the PHP client read. A shell_exec was used to send commands to the serial modules:

shell_exec("echo $command | netcat serialserver_ip port");

Dirty 👎

I was starting to use a MySQL database more and more; looking up the sensor names, historical data, etc. And shortly after I ditched the serial server and started using Python instead.

I also made a simple, frameless, version of this client — for WAP. Remember WAP? Wireless Application Protocol. Early day mobile phone web browsing 😄

Last commit 2023-02-05, with message: Add series for the rack box project.