Hacking O’Reilly’s “Fluent Conference 2013 ” video download page

The Fluent Conference 2013 videos were out and O’Reilly was offering a nice 50% discount to early purchasers. I went to their website and made the purchase. When I looked at the download page I realized that no videos were linked to my Dropbox account (like O’Reilly does when you purchase a book from them). I wanted to have a local copy of all videos to be able to watch them offline but O’Reilly only provides single download link per-video. There isn’t a “Download All” link and there isn’t any download manager available. My first reaction was to say -screw this- I’ll download them one by one. As I downloaded the first one I realized that not only the download speed was sub-par (1.5 megabytes per second) but I was being disconnected very often and the download had to start all over again (it wasn’t resuming). Not good.

Disclaimer: This post doesn’t describe any illegal activities but rather uses the term hacker in the way described in this Wikipedia article.

Plan B. Let the hacking begin.

I viewed the page source and realized that they were using jQuery. I also saw that the download URLs for all the videos were right there; there was no redirections or obfuscation of any kind. So after a couple tries using Firebug’s console, I came up with this jQuery:

This basically says “find all nodes with the format class and drill down to the second link. Give me the href for all these and print them out to the console”.

Great, now I had a list of the URLs for all the videos. I saved them to a plain text file. I now needed a method to download them all without me having to click, click, click…

wget

wget is very flexible when it comes to downloading stuff (which is a good thing since it was written to do that!). So this is what I tried:

The -i makes wget download files from list.txt. This worked great except that sometimes the download would stall exactly like it did when I was using the browser. Now, wget was designed to work well with crappy connections so it would retry a few times and start downloading again. But… sometimes it would just give up and would leave a half downloaded file. Restarting it again would cause the file to be overwritten so I added the -c (continue) param to prevent this and wget would pickup were it left off.

So far it was a great improvement: now the downloads would retry if failed and resume from where they left off. I only needed to solve the problem where wget would just give up sometimes. This happened very rarely so it wasn’t a big deal but still…

Bash

So I wrote this simple bash script that would just restart wget again if it stopped.

So even after the downloads had finished if the wget cycle was restarted, no significant bandwidth would wasted as wget was smart enough to not download stuff that it had already downloaded.

After the downloads finished I saw that the file names were horrible: ea4b8c913c44e_5813709.mp4?title=42_secrets-of-awesome-javascript-api-design-brandon-satrom.mp4. Yikes.

Enter PHP

I wrote this little script that does some quick and dirty string manipulation and renames all the videos it finds in a directory.

The script turned this:

ea4b8c913c44e_5813709.mp4?title=42_secrets-of-awesome-javascript-api-design-brandon-satrom.mp4

… into this:

42 – Secrets Of Awesome Javascript Api Design Brandon Satrom.mp4

Much better!

Learn how to read and write binary numbers in 5 minutes.

Over the years I’ve been asked to explain this to many people and most of the times as I was done explaining this, people would say “Huh, I thought it was harder!”. I think the reason is that when people are taught how to do this they are shown the math behind it and they immediately get bored and switch to “don’t know, don’t care” mode.  I’m not going to go over the math behind binary numbers because you can look that up anywhere but I’m going to show you a quick and dirty way that works every time.

The fact is, you can learn how to read and write binary numbers just by doing simple additions.

First of all, let’s build our mental cheat sheet. You will be writing a sequence of numbers starting from 1. The next number in the sequence will be the previous one times two. You will put each successive number to the left of the previous one. If you didn’t understand the previous two sentences don’t bother reading them back, just look at the picture and see if you can deduce the sequence because this is as hard as it gets. The rest is downhill:

binary_cheat_sheet

As you can see, if you take any number in the sequence and double it you get the number to the left. If you half your number, you get the number to your right.
Now write your binary number on top of your cheat sheet. Let’s take the number 10111 and write it on your cheat sheet. It’s OK if you have extra numbers on your cheat sheet, you can just leave them blank as we are not going to use them.

binary_10111

Now you add whatever numbers you have under every 1 and discard the numbers under every 0. Like this:

binary_23

It’s that simple.

Here are some other few examples if you want to try this out for yourself.

  1. 11
  2. 1001
  3. 10010
  4. 10000011

(Click the spoiler alert to show the answers)

Answers:
1) 3
2) 9
3) 18
4) 131

PCB Comparison: OSH Park vs. iTeadStudio

I took my 8 RGB LED Controller prototype and decided to try to make a PCB of it (my first attempt at a PCB, that is).

Most people at the Arduino forums had recommended I do this in EAGLE but they warned me that the learning curve was steep. So I tried doing it in Fritzing and failed miserably; couldn’t get the damn thing to do what I wanted. Then I gave EAGLE a try and thanks to Jeremy Blum’s tutorials could get a nice schematic done:

8 RGB LED Controller EAGLE schematic

Holy crap, that took me forever to do. When they say that EAGLE’s learning curve is steep they meant to say really f$#*@ing steep.

But the schematic is half the battle. After that you have to create the actual board layout which, again, took me forever to do. After many hours of getting cryptic errors and solving them by reading random forums, I got this:

8 RGB LED Controller EAGLE PCB

So far so good. Now I had to send them out to get printed.

A lot of people recommended iTeadStudio (a Chinese PCB manufacturer) and some other recommended OSH Park (US based), so I ordered a set of PCBs of each well aware that my design could easily be flawed and end up with a bunch of duds. For the same money OSH Park gives you 3 boards and iTeadStudio gives you 10. I was very curious to see what the quality difference was.

Note: A week after sending my order to iTeadStudio they wrote back saying that my design was 2mm over the limits of my chosen PCB size and they put my order on hold until I sent them the mind boggling amount of $2.90 extra. Yes, they put the order on hold for less than $3!

Three weeks later both orders arrived just a couple days apart. And this is what I got:

OSH Park vs iTeadStudio

Even to the untrained eye (like mine) there are a few differences worth noting. Both providers happily ignored the boundaries I set on my PCB. Note that the word “Controller” is cutoff on both boards (top right). You might be going back to the EAGLE board screenshot above and seeing that there is no word “Controller” there and you’d be right; I took that screenshot a few minutes ago from a more recent version of the board, but trust me, that word was well withing the boundaries of the board. Also note how iTeadStudio’s is much taller and also better trimmed while OSH Park’s is shorter and you can clearly see the marks where they cut the PCB off the neighboring boards.
Another thing to note is that OSH Parks’ board has “golden” pads while iTeadStudio’s has silver pads. I have no idea (and don’t really care) what the material is but the only thing I can say is that the OSH Park board required much less heat for the solder to “attach” to the pads so I guess it was easier to work with. The overall glossy purple finish looks way cooler too.
A funny thing on iTeadStudio: I paid for 10 boards but they’ve sent me 13. After reading around a bit I learned that this is not uncommon for them. Cool, I’ve got 3 more boards to mess with.

So I put it together and after a few minutes of soldering, got this:

8 RGB LED Controller PCB

Those SMD resistors were a bitch to solder, that’s for sure!

Now for the maiden voyage. I hooked it up with some LEDs and the Arduino fully programmed and ready to go… hmmm… why is NOTHING happenning? It didn’t work at all. I checked and double checked everything and every single cable was working correctly. So I soldered the OSH Park version of the board and tried it. Exact same results! Everything was pointing at a fail of epic proportions on my PCB design. So I gave up and went at it the next day.

Armed with my wimpy RadioShack multimeter and the prototype in hand, I soon realized that I had screwed up 2 things on my design:

  1. That “Sensor” label I put there to be able to hook up a button? It was wrong but easily solved by adding just a jumper cable from +5V to “Sensor”.
  2. I must have been really tired when I labeled the screw terminals as I reversed the ST_CP and DS on the shift registers. Nice.

So I hooked up everything like I should have to begin with and, oh yes, the damned thing started to work.

Next step, make this into an Arduino shield!

8 RGB LED Controller

UPDATE: I’ve now made a custom PCB out of this.

This is my first experiment with Arduino. Like most people starting out with Arduino, I wanted to make stuff blink! So after finishing up the Arduino Starter Project Book I started hooking up LEDs to my Uno’s outputs. I very quickly ran out of outputs and a quick search through the Arduino Forum led me to the wonders of shift registers. These are basically serial to parallel ICs that allow you to just use three pins on the Arduino to control eight outputs. You can daisy chain them like I did and if you do, you can control many more outputs yet still using three pins to control them. I few days later, my SparkFun order had arrived and it was time to test my spanking new 74HC595N set.

Ingredients:

  • 3 x 74HC595N shift registers.
  • 8 x RGB LEDs, diffused.
  • 8 x 200Ω resistors
  • Lots of jumper cables
  • 1 x Solderable PC Breadboard.

After some trial and error, I arrived at this. The gray cables are telephone cables that connect the controller to the LEDs.

8 RGB LED Controller prototype

Yikes, that looks ugly. I soon realized that I shouldn’t have soldered the LED cables to the board as it makes it very impractical to repair any LEDs or reuse parts. Even though it looks flimsy, so far it’s still in one piece.

I’ve put together a video that shows some of the build process and also the whole thing put together and working. At the end of the video you can see the presets that come with Elco Jacobs’ awesome ShiftPWM library which is what I used to control the LEDs.

 

Long exposure shots

I recently bought a shutter release for my Canon 450D. This allows me to use the BULB mode on my camera and shoot very long exposures. I took it for a spin yesterday and this is what I came up with. These were taken in Boston, MA where Mass Ave meets Westland Ave. Some of these were exposed up to two minutes. I  didn’t have the patience to try longer times, but maybe next time…

Boston - Mass Ave

Boston - Christian Science Center

 

Boston - Christian Science Center

Boston - Christian Science Center

Boston - Christian Science Center