INTRODUCTION
Few months ago
Andrej Karpathy wrote a
great blog post
about recurrent neural networks. He explained how these networks work
and implemented a character-level RNN language model which learns to
generate Paul Graham essays, Shakespeare works, Wikipedia articles,
LaTeX articles and even C++ code of Linux kernel. He also released the
code of this RNN network on
Github.
It
has been a while since I have experimented with RNNs. At the time I
found RNNs difficult to train and did not pursue any further. Well,
all that has changed in the last year or so. I installed Andrej's
char-rnn package
from Github in less than 10 minutes on my Linux laptop using
instructions on the Readme.md file. I tested the installation by
training the RNN with the Shakespeare's collected texts provided as part
of the package.
If you have GPU graphics card (like NVIDIA
Titan) the training goes much faster. I did not have this so I let the
training run in the background for over 24 hours on my Lenovo X301
laptop . Looking the results the RNN indeed learned to output
Shakespeare like language as Andrej explains in his blog post. It
certainly took me more than 24 hours to learn English language and I
never learned to write dialogue like Shakespeare. Please note that RNN
was a "
tabula rasa" so it had to learn everything one character at the time -
this was pretty amazing result!
I decided to do an experiment to find out if this RNN technology could be used to build a ham radio robot.
TRAINING A HAM RADIO ROBOT
The
robot would have to learn how people make CW QSOs in real life. I
collected some 10,000 lines of examples of ham radio CW QSOs from
various sources. Some examples were complete QSOs, some were short
contest style exchanges and some just calling CQ. The quality of the
language model depends on the amount of examples in the training file.
To
do this properly I would need at least a few megabytes of examples, but
I found only about 200 kBytes after a few minutes of Internet search. I
copied this material into a single file named "input.txt" in data/QSOs
directory and started the training with the following command:
th train.lua -data_dir "data/QSOs" -gpuid -1
RNN
reads the input data, builds a dictionary, configures the neural
network and parameters and starts the training. During training the
software outputs examples in the cv directory. This allows you to see
how RNN learns from the provided material.
After only 12 epochs
the RNN had already learned some basic concepts like call signs and
basic words, but it was not enough training rounds to learn English or
proper syntax of CW QSO.
GG3QXP DE WA1AXP/M K
RARR
WIS DE M8BPY RR NAMME FDO RR0S ANOEN CUX/I AO RNEL SNETSL CL ATRSO
IANLOED UR RV SI MEE DO RU @ UR RSR IX NUOR IS QS IN WIEZ DE QEMUM/7 W
After
25 epochs the RNN had reached to the contest section (November
Sweepstakes) of the input file. Note that the call signs below (like
VE7LP or K2XC) were not included in the
original training file. RNN has learned the proper call sign syntax from given examples and is generating this text below:
VE7LP 314 Q 52 SK=
K2XC 153 M 68 WNY=
K8CSC 285 A 15 MI=
AE5MX 341 Q 47 SB=
AE7J 419 A 06 WWA=
K5RMC 137 M 43 OK=
KB2QUT 34 U 21 NNY=
K9MIP 120 M 08 WI=
Wow...RNN has also learned
November Sweepstakes exchange from provided examples. Even the
ARRL sections are
correct. Note that RNN is generating these based on learned rules, not
just copying provided examples. All this by reading the input text file
one character at the time.
After 39 epochs RNN has learned many
CW acronyms, RST report and some basic QSO structure, though there is
still a lot of nonsense in the output. RNN talks about name, QTH,
weather (WX) and even RIG and antennas:
GUDXTEN/8
IN = NAME HR IS ED ED QTH IS CHAPOTON FNCI HOO OO DINED MIAD VALT W FO
FON CUR DS MS ES TOT FER CL IW QSO OB ULLOO = QRHPO DE LOOFSD
SORK/ISTO= = = RIG HR IS LACKORI WAZH PA WANFOO = WX WR = 2IT WINLE WOOD
DES UP 55 FE HW? + MJ1GJO DE MJ3ASA K
GUD DX ES 73 G8XFO DE 2E3CUD/9RP @
KC6XQ DE M5WMM/M DE M1TGL/M K
W63ED DE M5YUE
VVV VVV
CQ CQ CQ DE WA1NX/WA50 WB4AJH/6 KC0AHH K1 WAJH K
WA3JRC DE W4DD/5MM DE KC3GJJ/8 K
GV8SDE DE 2I8APZ GD ENZ/QRP GD3BOB
G1KHC DE G3ECQ/QCP M7Y
VVV VVVVV
CQ CQ CQ DE W3ARJ/0 W11IA DE M9TFO/P WA2MJH/4 K
1BJ0 WA3BH DE MJ1GDS MJ6XW K
MW6PJP/M
DE MW6QIC UR RST HR 459 W QSB M5YES DE 2E6AO QRS 6NT QSO D = RIG HR IS
K70 WNTN = NAME HR IS JIM JEC QTH IN COLLA NCME HO FEL PU DS MS ES
2INTOOBA MCONTS = = UR RRTR MNNI W IS HOX WBOO ISTEL INYD = =
GUD RIS HL W0 IN W WOME DEN UTD PO P =
At
this point RNN has seen the complete training material only 39 times.
It has figured out many rules (like ending its turn on "K") but it has
still problems with vocabulary, how to use call signs properly, etc.
After 50 epochs the RNN had already mastered idioms such as TNX FER NICE QSO CU AGN and structure resembles a real QSO:
KC2RD/4 DE W1GJV K
W1ARD DE KC2RD/3RO UR RST HR 459 W QSB GD3DIA DE GD9GEN @ CD GG2RD/9 DE W1GRV/4 DE GU5TCH/MM R TNX FER NICE QSO CU AGN
M2YXT DE GD5FM UR RST HR 529 W QSB W1GRD DE W1GRR RR K
GG TI TE UR 33
IWAMO DE WA6EN
KC2X DE W1YDH KE9NZE/0 OL TU
UR RST HR 309 W QSB = NAME HR IS AANNY WAVEL FNH COTE TNX QST
= UR 7S PR = UR RST HR 599 W QSB = HR VY NERVOUS D DE MYUE USD 1S =
NAME HR IS DI EESTY == RIG HR IS HEATH 71 INTO A NME HR IS VILL HW?
2E9AAT DE GW6QI UR TS TAX DEL NAME H5 UE EU 539 FE KHHJ RS 2E MES LANNY =
QRY = NAME HR IS ED ED QTH IS PARD VORETP
You
can also see that some parts (like NAME HR ) are repeating multiple
times. This was also noted by Andrej in his experiments. Since the
training is done one letter at the time, and not word by word the RNN
doesn't really get the context of these phrases.
PRACTICAL APPLICATIONS
This
kind of ability to provide predictive text based on language models is
widely used in many Internet services. When you type letters into Google
search bar it will provide you alternatives based on prediction that
has been learned from many other search phrases. See figure 1 below.
 |
| Figure 1. Predictive search bar |
In
the same manner RNN could provide a prediction based on characters
entered so far and what it has learned from previous materials. This
would be a useful feature for example in a Morse decoder. Also, building
a system that would be able to respond semi-intelligently for example
in a contest situation seems also feasible based on this experiment.
However,
there is a paradigm shift when we start using Machine Learning
algorithms. In traditional programming you write a program that uses
input data to come up with output data. In Machine Learning you provide
both input data and output data and computer creates a program (aka
model) that is then used to make predictions. See figure 2. below to
illustrate this.
 |
| Figure 2. Machine Learning paradigm shift |
To
build a ham radio robot we need to start by defining the input data and
expected output data. Then we need to collect large amount of examples
that will then be used to train the model. Once the model is able to
accurately predict correct output you can then embed it into the overall
system. Some systems will continuously learn and update the model on
the fly.
In the case of ham radio robot we could focus on
automating contest QSOs since the structure and syntax is well defined.
In the experiment above RNN learned the rules by seeing the examples
only 25 times. So the system could be monitoring a frequency, perhaps
sending CQ TEST DE <MYCALL> or something similar. Once it
receives a response it would then generate the output using the learned
rules and would wait for acknowledgement and log a QSO.
If the
training material covers enough "real life" cases, such as missed
letters in call signs, out of sequence replies, non-standard responses
etc. the ham radio robot would learn to act like human operator and
quickly resolve the issue. No extra programming needed, just enough
training material to cover these cases.
CONCLUSIONS
Recurrent
Neural Network (RNN) is a powerful technology to learn sequences and to
build complex language models. A simple 100+ line program is able to
learn complex rules and syntax of ham radio QSOs in less than 50 epochs
when presented only a small number of examples ( < 200 kBytes of
text).
Building a ham radio robot to operate a contest station
seems to be within reach using normal computers. The missing piece is
to have enough real world training material and to figure out an optimal
neural network configuration to learn how to work with human CW
operators. With the recent advances of deep learning and RNNs this
seems an easier problem than for example trying to build an automatic
speech recognition system.
Περισότερα:
