Tuesday, December 16, 2014

Converting hex to binary in the brain

This is a tutorial on hex which is very useful if you are ever going to read low-level code or program low-level things like network protocols or microcontrollers. I use a real project that I worked on to showcase all this, namely a matrix of 9 LEDs.

led_fisheye_300

You should be able and understand why people put hex in the code instead of raw binary (if it exists for that programming language). There are very specific reasons for doing this and since converting from hex to binary is so damn easy, there is no excuse for you to not be able and do it in your brain.

Binary and LED patterns



I was building a trivial LED matrix the other day for an MBED microcontroller (think Arduino-like). The usual problem is that my brain is faulty so I do all sorts of things in the wrong way. I take this blog as the opportunity to make up for what I learn just to make sure that I won't forget (and ofcourse to teach others if they are interested).

So my task was to achieve some patterns with 9 LEDs. Notice that it doesn't matter how the microcontroller was connected etc. since here I am only dealing with how bits and bytes fit into low level programming. My first pattern was a rotating star that you can see below:

rotating-star

This star is made out of two LED patterns: a star and a cross.

text3997

The 1s and 0s simply mean if the LED at a position should be turned on or off. Now, when we're dealing with low level things the minimum unit of information that can be sent is 1 byte (= 8 bits). In our case we have 9 LEDs so we need however a minimum of 2 bytes (= 16 bits). The above examples become the below binaries:

Star:  0000000101010101
Cross: 0000000010111010


Now, the problem is that when we deal with low level programming, most low level languages (C, C++ etc.) don't let you write numbers as binary in your code. You can't write printf(0b101) for example. You need separate libraries if you want to do that and that would be fine for our case. But imagine if there was a matrix of 100 LEDs. Someone reading printf(001001010101101010101010101010101110101001011100101) would just get lost in the 0s and 1s. That's one of the big reasons hex is used - it's super minimal.

Binaries as integers



At first when I wanted to create a star, I simply converted each binary to an integer and just put it in my code. Below you can see a simplified version of my code.

[sourcecode lang="C" gutter="False"]
..
#define STAR 341
#define CROSS 186

int main() {
while (1) {
leds.write(CROSS)
sleep(1)
leds.write(STAR)
sleep(1)
}
}
[/sourcecode]

The way I found those integers was by simply using Python. It is a rather trivial procedure:
[sourcecode lang="Python" gutter="False"]
>>> 0b101010101
341
>>> 0b10111010
186
[/sourcecode]

Notice that I omit the extra 0s since they don't add any value just like 000150 is always going to be 150 no matter how many zeros you add at front.

Binaries as hex



The code I used, worked fine. The problem with this solution is that it's impossible to have a clue what an integer is in binary - and when we deal with low-level programming that matters most of the times. In our case for example each single 1 and 0 controls one LED. Being able to figure out fast the binary of a number in this case is very important.

For example say you find the code snippet below somewhere:

#define STAR1 341
#define STAR2 186


How can you tell if it's the STAR1 or STAR2 that looks like an 'X'? It's just impossible. And what if there were many more stars or if the LED matrix was huge? Then it would be a nightmare to understand the code. That's where hex comes in handy.

The good thing with hex is that someone can see the hex and decode it to binary in his brain. So say we had the above example but instead of integers had hex values:

#define STAR1 0x155
#define STAR2 0xba


A skilled programmer would directly see 0001 0101 0101 and 0000 1011 1010 with no effort. And he wouldn't either need to decode the whole number to find out. Watching just the last hex digit of each STAR would give him (or us) a hint about which STAR is which.

It's time we become that skilled programmer, don't you think?

Hex to binary in da brain



Fortunately it is very simple to convert hex to binary in the brain. You simply have to understand that each hex number is made out of 4 bits since we need a max of 4 bits to represent the largest number in base 16 (which is the character 'F'). So 0xF is 0b1111. (Notice that putting 0x in front denotes that the number is in hexadecimal represation and putting 0b denotes the binary representation accordingly.)

The procedure of binarizing a hex is simple:


  1. Find the binary of each hex character

  2. Place 0s in front of each binary (from above) so we always have 4 digits

  3. Squeeze them all together as if they were strings



So for example:

F   is       1111
5 is 0110
FF is 1111 1111
55 is 0110 0110
5F is 0110 1111
F5 is 1111 0110


Hopefully you get the hang of it. The question is.. what happens if we have 0x102? This might seem tricky since we get three very simple binaries: 1, 0 and 10. But as I said, if you add the 0s in front before you squeeze them together, you should get the correct value - in this case 1 0000 0010!

Also you need to memorise a few things to be able and do all this. I have written the bare minimum below:

Binary     Decimal
1 = 1
10 = 2
100 = 4
1000 = 8
1010 = A
1111 = F


Then it's quite easy to find in brain all the rest. For example to find the binary of B we can simply think that A is 1010, and that since B is just one step ahead, we add 1 to it and thus get 1011. Or to find 5 we simply have to add 1 to 4 which becomes 100+1=101. And so on.

This should also make it clear what the command chmod 777 in Linux does.

Big fat hex stars



The below is more like an exercise to test what we've learned. It should be rather simple to find the hex of the star below.

star_7x7

It might seem overwhelming, but the only thing you need to do is go in groups of 4s and write down each hex value.

Grouping in 4bit groups:
star_7x7_marked

Decoding the above becomes 8388A08388A0 which is WRONG.

Row 1: 8 3..
..
Row 7: ..A 0 (and a remaining 1?!)


This was actually a trap to teach you the hard way that we should always start from the last digit. In this case in the end we are in a situation where we have an orphan digit 1. We can't work with that since we need 4 digits to make a hex number.

The right way is to always start from the end. This is for all numbers no matter if they are represented in binary, octal, hex, decimal or whatever - as long as they are numbers, always start from the last digit and you'll do fine. The reason is that when you finally get to the last number you can add as many zeros as you like (or need) without altering the value of the whole thing.

So the correct grouping is this (starting from bottom-right):
star_7x7_marked_right

And then we just start from the bottom and get 1051141051141! Notice that in the end we again have a single 1 (at the top left this time), but this time we can add as many zeroes as we want since adding zeros in front of a number doesn't change its vallue.

We can also validate what we got with Python:
[sourcecode lang="Python" gutter="False"]
>>> bin(0x1051141051141)
'0b1000001010001000101000001000001010001000101000001'
[/sourcecode]

Thursday, December 11, 2014

Jenkins' time is over!

If you go to any serious company you will find that they have automated tests running for every new update in the codebase of the programs they develop. This assures that every random thing some developer fixes, doesn't break something else in the program. In most cases you will find Jenkins or some of the commercial solutions like Travis and CircleCI for this.

As a metro-sexual hipster developer however you want something that is fast to setup, easy to use and minimal. To be fair there is a huge amount of open-source free continuous integration (CI) programs that do this. On my recent hunt however I found one that shines amongst them: StriderCD.



The big advantages of Strider:


  • Stable

  • Beautiful intuitive interface

  • Integration with Github

  • Small enough to hack (they even have a guide on how to hack it!)

  • Plugin API

  • Helpful active community!



In this tutorial I will setup Strider so that it works even if you sit behind 9.000 firewalls in your company. I will also set it up so that you can move it around and reuse it in the future without the hustle of setting up things from the beginning. For this I will use Docker ofcourse, the hipster's French wrench. I will also make it so that you can run other containers inside the container where Strider runs. This is helpful if you want to run tests inside their own container for example.

Building Strider



To be sure that you have the latest Strider it is wise that we build everything from scratch. I have created a Dockerfile that will make an image with the latest Strider and setup everything up so that you can use it with Docker.

In case you don't have Docker:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys     36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install -y lxc-docker


Then we can just build Strider:

git clone https://github.com/Pithikos/Dockerfiles
cd Dockerfiles/strider-docker
./preprocess
docker build -t strider:barebones .


preprocess is a simple shell script I made that will edit the Dockerfile so that in the next step where we build it, it installs the same Docker version as the one outside of the container. This is critical since if the Docker inside the container is different from the one outside of it, we won't be able to run Docker inside the Strider container.

Running Strider



At this point we have a Docker image named strider:barebones which has a minimal but powerful setup for Strider. It's time we run it:

ID=$(nohup docker run -d --privileged -v $PWD/data/db:/data/db -p 3000:3000 strider:barebones)


This will run the container with the database stored at $PWD/data/db. That way if for whatever reason the container stops running, we have all the user accounts, projects, etc. intact. Notice also that I use the --privileged flag. This is merely so that we can run containers in containers.

To verify that everything works open your browser at http://localhost:3000 and you should see the login page. Notice that you won't be able to login to Strider at this point since my Dockerfile recipy keeps things minimal - that means no user accounts by default. Adding accounts is super-easy and you don't even have to restart anything so that won't be a problem.

Screenshot

Adding an account



In order to add an administrator account to Strider you just have to access the running container and create a user from within. If you use an older Docker you can use my tool docker-enter. If you run Docker 1.3+ then you can use docker exec.

docker-enter way:

sudo docker-enter $ID
strider addUser
..
exit


docker exec way:

docker exec -it $ID strider addUser


Follow the instructions to make a new account. Then just exit the running image and just refresh the webpage http://localhost:3000 (no need to restart the running image). You should be able and login in to Strider with the credentials you just gave!

Saving for the future!



Now, notice that all configurations, projects, user accounts, etc. are all stored in the folder db/data. In order to move the whole thing somewhere you simply move the folder to the computer you want and run strider:barebones as we did before. Ofcourse you will need to rebuild the whole thing as before as well or you can either save it to the cloud:

docker commit $ID username/strider:barebones
docker push username/strider:barebones


Remember to replace username with your own username at the Docker registry. Once all this is done, you can run directly strider from any computer. The image will be downloaded automatically.

docker run -d --privileged -v $PWD/data/db:/data/db -p 3000:3000 username/strider:barebones


If data/db is not at the directory where you try to run Strider then a fresh database instance will be used.