Programming Collective Intelligence

Programming Collective IntelligenceProgramming Collective
Intelligence

  • Building Smart Web 2.0 Applications
  • Toby Segaran
  • Web Applications

If you've ever wondered how web-sites can suggest other books, music or movies that you might enjoy then this book will tell you. It's packed with code for grouping, clustering, filtering and analysing information. If you use Python and you either understand, or want to learn more, about Bayesian and Decision Tree Classifiers, Neural Networks, Support-Vector Machines, k-Nearest Neighbours, Clustering, Multidimensional Scaling, Non-Negative Matrix Factorisation and Optimisation then this book could well be for you.

But, if you're like me and use Ruby and aren't a maths wizard then most of the content may well pass at great height over your cranium. Yes, I did read it all, if only to try and glean more of an understanding of the techniques involved. It started well, discovering groupings in RSS feeds, then the concepts became trickier to grasp. I think the only thing that kept me going was one of the last chapters on evolving intelligence. Being a developer of software I've often wondered how you can write something, given specific rules, that can develop it's own intelligence. Especially when these things can play games with themselves and learn as they go.

The only chapter that I'm certainly going to re-read and look into is the one on discovering groups. My idea is to add a related posts, and possibly even unrelated posts, section under each post on this web-site. From what the book says this would look at the text of each post and find ones with similar words and counts of words. I may even be able to plot a dendrogram showing clusters of topics.


Secure Passwords

Nowadays everyone has far too many passwords to remember. Personal ones for computers, server passwords, online shopping passwords, online accounts for paying bills as well as all the social websites that everyone signs up for, but then doesn’t use. Creating secure passwords is a problem. You can’t just use dictionary words because of brute force cracking and you shouldn’t use the same password for everything… just in case.

There are a few software solutions, such as 1Password, that I've tried but I didn’t like the fact that it generated secure passwords that I couldn’t use elsewhere, like on my work laptop.

Here is the technique that I use for generating secure passwords.

Qwerty Keyboard

Select a memorable phrase or a name, the more uppercase letters the better, and look at keys on the keyboard. Instead of typing the name as it is use the key above and to the left. If you can touch type then it is just a case of using ‘R’ and ‘U’ as the index keys instead of ‘F’ and ‘J’.

For example :–

Barack Obama becomes Gq4qdi (gqjq
Britney Spears becomes G485h36 W03q4w
Alice Cooper becomes Qo8d3 D99034

I've left the spaces in to make it readable but you get the idea.

The only snag, that I've found, is using it to enter passwords into Mobile Safari because the numeric keys aren’t displayed on the same screen, to save space.

So you have generated a very secure password using a really simple encryption technique.


All Change

All ChangeAll Change

Yes, it’s all change… again.

For the first time, in a long time, this site has switched engines. Previously it used WordPress but now it’s using a Ruby on Rails blogging application called Enki. I haven’t stopped using WordPress entirely as the website for my Uncle Les is still running very well on the latest 2.7 release.

I had spent a month or so trying to write my own from scratch, and did manage to get something working, but I kept finding that I was struggling to get the basics polished off. Enter Enki, which is just a barebones blogging app that you can build upon.

So, I've removed (or at least commented out) the comments section of code. I might resurrect commenting on the site but, as I've found previously, it’s much more trouble than it’s worth. I've switched to Markdown for markup instead of textile. I don’t quite know why, just that I read http://daringfireball.net/ all the time. The only other ‘behind the scenes’ changes are a switch to FeedBurner for the RSS feed (so update your reader) and I've used Google Custom Search in a similar way to the WordPress set-up that I had.

My Advanced Rails Recipes book came in really handy when it came to generating a sitemap, creating custom error pages and a different maintenance page. I’ll be using more of the recipes in that book in the future.

The style of it all is just really simple and is based on the 960 Grid System. I'm only really using the grid for the basic layout at the moment, but I shall use more of it as I add to the site. What are the chances that it becomes as cluttered as the last in no time.

I still have a few things to fix. The book reviews are still all in HTML because Markdown thinks that because it contains a ‘div’ that it isn’t needed. Rspec tests don’t pass because I've changed some code to get the book reviews to use a different partial etc.

Just consider it a work in progress which can be found in all it’s glory at github.com.


iPod touch 2g

iPod touch 2giPod touch 2g

Well, it didn’t take me long after the Stevenote to order an iPod touch second generation from the online Apple store. It has been a long hard struggle… but it’s been worth it.

I did originally think about buying an iPhone, even before the 3g model came out, because it would save me having to carry a phone and an iPod nano. With the phone I'd get phone calls, text messages, the built-in camera and internet access on the go for £35 a month minimum. I even waited for the 3g release and news that O2 may have a pay as you go plan for the phone. That news was released the other week and really didn’t seem like a good deal. Available from September 16 it would cost £399.99 for the 16GB version, unlimited browsing and Wi-Fi for 12 months, after that £10 per month. Then there is a minimum top-up of £10, even if you don’t use the phone.

Besides, I don’t use the phone. I've been with Virgin Mobile since 2000 and have only topped up the phone twice. So when Steve Jobs announced the new iPod touch with 16GB for £219 I decided to get one. Even then I was undecided about getting the 32GB, but my iPod nano, which has 8GB, is only half full with podcasts so I couldn’t see myself filling it to capacity.

This new iPod touch is really nice, reassuringly hefty in the palm, thanks to the solid metal back and the glass touch screen. The interface design really is just a joy to behold. Everything just zooms, spins and scrolls with ease. Remarkable how they have managed to get everything to work with the minimum amount of buttons.

Naturally I've installed a few applications :– Air Sharing, BubbleWrap, Cro-Mag Rally, Dactyl, Enigmo, Hanoi, Instapaper Pro, iWik, Koi Pond, NetNewsWire, OmniFocus, Remote, Tap Tap Revenge and Twitterrific. I've had a little dabble with all of them but NetNewsWire, for reading my news feeds and syncing with my iMac, was a must and Instapaper Pro was a no brainer for reading the web pages away from my desk.

That’s really all I'm going to use it for, surfing the web, checking emails, news feeds and reading web pages. I have put some TV shows, music, podcasts, photos, music videos, even PDFs on it but I’ll still use my nano for listening to podcasts when I'm walking, cleaning or cooking because it’s just the right size for my top pocket.

  • Posted on Monday, 15 September 2008
  • Tagged with toys, apple, web

Klein Bottle

Until lunchtime today I didn’t know what one was either.

I've been watching the TED Talks on my AppleTV, I still have about 20 of the normal video quality ones to watch before I start on the HD variations. Because they’re only 18 minutes long, I watch a few eating breakfast, then a couple at lunchtime and maybe one or two whilst eating in the evening. I am quite the fan.

Today I watched this talk by Clifford Stoll

It is such a wonderful and amazing talk made by someone who looks like Dr. Emmett Brown but occasionally has the voice of Emo Philips. No disrespect intended to anyone concerned.

Cliff StollCliff Stoll
© Cliff Stoll 2008

As Cliff mentions in the talk he makes klein bottles. I certainly hadn’t heard of them but I had heard about Moöbius and his loop with only one edge and one side. So later in the afternoon, and certainly not on company time, I found his website and ordered a big klein bottle.

In the evening I received an email from Cliff Stoll (Chief Bottle Washer) thanking me for my order and saying that he took some photos, which would arrive shortly. A couple of minutes later an email with eight photos attached arrived in my in-box showing the klein bottle, the bottle in bubble wrap, the bottle in a box with packing quavers around it and one with Cliff holding the taped and labelled box.

I've bought many many items via the internet, and I've had a few email chats with the sellers, but I've never had such a fantastic experience. And I haven’t even received the bottle yet, that pleasure is between a week and ten days away.

For someone selling something that they've produced, and in limited quantities, this would be a remarkable idea. To see the person who has made the product and see it being packaged isn’t such a huge drain on resources, and it adds so much to the overall experience.


Programming Amazon Web Services

Programming Amazon Web Services: S3, EC2, SQS, FPS, and SimpleDB Programming Amazon Web Services:
S3, EC2, SQS, FPS, and SimpleDB

  • S3, EC2, SQS, FPS, and SimpleDB
  • James Murty
  • Web Programming

Who would have thought, when Amazon started selling books back in the late nineties, that you'd be able to 'rent', for want of a better word, computing time on their servers. And to store as many files as you wanted on their hard-drives. That you could, as many companies have, run a good chunk of your infrastructure in the 'cloud', without having to buy servers and have them installed at a secure facility with fail-safe systems.

I bought this book after using Amazon S3 to store my iPhoto files back in January. Then, and still now, I use a set of Ruby scripts called S3Sync to synchronise files from my iMac to S3. That is more-or-less all I use S3 for apart from automatic backups from my web-server every night. I did watch a screencast done by someone who had written Capistrano recipes that deploy a Rails application to EC2, which was just mindblowing... and a little depressing, as I thought that I should be doing that kind of stuff at work.

So I was interested enough to want to learn more. I was also hoping that it would cover Amazon AWS Search, to retrieve product information, but it doesn't touch on that at all. As the title suggests it covers S3, EC2, SQS, FPS and SimpleDB. Don't think that I've read it all as it is close to 600 pages and, as you can imagine gets very technical. Luckily all the example code is written in Ruby, so I did have some understanding of what the code was going to do.

At least now I have a basic understanding of what is possible with these services. I would be interested in setting up an EC2 instance to run a Rails application, just to see what is involved.

The book is certain very thorough and, having only been published in March 2008, is relatively up to date, but as Amazon change and improve their offerings how relevant will it be next year.


Back to Belkin

The router, supplied free by O2 when I signed up to them for broadband, has proved a little problematic to say the least. Because it’s just on a shelf under the desk I could hear it clicking occasionally. This was when it was resetting itself, for whatever reason. If I was browsing web-sites I'd have to wait a minute or two before I could continue. Not ideal, so I've switched back to my Belkin F5D7633 wireless router. Here are my settings if anyone else want’s to switch to a more dependable router :–

  • VPI/VCI – 0/101
  • Connection Type – Dynamic IP (1483 Bridge)
  • Encapsulation – LLC
  • DNS – Auto
  • IP Address – Automatically Assigned
  • SSID – changed from the default
  • Broadcast SSID – Off
  • Security Mode – WPA/WPA2
  • IP Address – changed from the default

The SSID has been changed, and switched off, to stop anyone trying to access the router. That’s using the principle that if they can’t find it they can’t hack it. I've also switched to WPA from WEP for extra security.

Most of this information I found in the forums at Think Broadband.

  • Posted on Saturday, 22 March 2008
  • Tagged with web

Words fail me...

I received an email from Tiscali Provisioning this morning saying… well it’s all a little difficult to make out as the subject and the three sentences are crammed with codes and references. Essentially they say that the customer, me, is still active with another ISP and that BTOR(!) is currently in Cancellation Pending state, whatever that means. And that we, them, need to monitor the order for the next 24 hours.

Obviously I have replied as I certainly don’t want them cocking-up, a technical term, my excellent internet connection with O2.

It is good to see that Tiscali are about as on the ball as Pipex were. This sorry saga started on the 4th of March so I could have been without a connection since then, if I had been stupid enough to stay with them.

My advice to anyone still with Pipex is if your connection is fine, you consistently have fast download speeds, and you are happy with the service then stay. If you aren’t happy then switch as soon as you can but expect problems like I had. But 10meg download speeds, unlimited and £10 a month cheaper it has to be worth switching.

  • Posted on Saturday, 22 March 2008
  • Tagged with moans, web

O2 Oh yeah!!

Yes, yesterday I received an SMS message saying that O2 had connected my broadband that it was now safe to plug in the router. Why you can’t do it beforehand I don’t know. I just did what they told me.

After setting up the router, that is just plugging it in, I went straight to think broadband and did a speed test. 10 Mbps down and 1.5 Mbps up, which probably what you would expect from a 16meg line. On an 8meg line with Pipex the fastest down was just 3Mbps, but possibly I only ever tested the speed if it was really slow.

So the fun began, downloading email, news feeds and the podcasts that I'd missed over the last week. Then it was just a case of setting up the wireless connection for my Apple TV and the Dell laptop that I have for work. All connected to the O2 router fine.

I hit Send/Receive in Microshite® Outlook 2000 on the laptop and downloaded the three waiting emails. Then, for some reason, I had to send an email, no doubt to let the world know that I was back and online. The indicator kept saying that it was sending, sending, sending, failed. I hadn’t changed anything in Outlook. It had worked fine at my sisters house the other day. Try again… and again. Nothing. I phone the company tech guy and he suggests deleting the account and adding it again. I take screen shots of the settings, delete the account and add it again. Still nothing. SMTP was timing out, obviously when trying to send. I switch off the firewall in the router and try again… nothing. I delete the dial-up account that Pipex had given me that didn’t work… nothing. I change the sequence of the wireless connections, my sister’s being the top one… still nothing. But if I connect to the office I can send email fine. I install the O2 software from the disc that came with the router, just in case… nothing. I start looking at the firewall settings in the router again… nothing. Not a problem, I’ll switch to the Belkin router tomorrow.

This morning I reset the Belkin router and try to get it working with O2. You can see why O2 supply and support their own router. All this is tricky enough if know a little about how this stuff works, but if you don’t you’ll be screwed. I restored to factory settings and spent about an hour trying to get it to connect to O2 with no luck. Annoyed, very annoyed. Considering that my iMac worked fine with the O2 router and that my laptop worked fine with the O2 router… except this little problem sending email. I put the O2 router back tried to do some work. But a little problem like this just nags and nags. You think that it’s just going to be a little setting or a check-box that you've overlooked. So I downloaded Thunderbird. God knows why a software company should still be using Microshite® Office 2000 anyway. I set up my pop account and it sent email fine. I imported my contacts and old emails from Outlook and haven’t looked back since. It’s all working fine.

I just hate the fact that I wasted over a day trying to get something that simple to work, even if I was being paid for it. Good old Microshite®!


Torture by Pipex...

27/02/2008

The story so far… after the, so called, engineering on the line I receive another MAC code, which, coincidentally is exactly the same as the previous 3. I phone Pipex and they say that they can now see a cancel order on the line, which is good news as they couldn’t before, but with the MAC code due to expire there wouldn’t be enough time for O2 to use it and get me transferred over. They suggest phoning again to request another MAC code.

04/03/2008

Mid morning and my internet connection dies. I try re-booting the router and wait until after lunch before phoning Pipex, just in case it’s a temporary thing. The first people I call are the MAC code guys and I ask if the fact that I have no internet connection is something to do with the MAC code expiring the bloke on the other end of the line says, and this is important, “No, we would never cut you off!” I then phone the technical support line, they do a test on the line and do confirm that there is no internet traffic on the line. Voice traffic, fine, no internet. The suggest plugging the router into the test socket, trying another micro-filter, trying another router. All this I tried. Still nothing so I phone them back with my findings and they say that they’ll log it with their engineers who could take between 1 and 6 business days to fix the problem. Not good news for someone who WFH and uses his internet all the time when he’s not working.

05/03/2008

Luckily my sister, brother-in-law and niece have a wireless broadband connection. So I walk around to their house and work from their sofa for the afternoon.

06/03/2008

Phoned Pipex technical support again. Tried for about 20 minutes to get through and was told more than once by a recorded voice that the line was unavailable. Click. Finally spoke to a human he said it was all to do with the Tiscali re-engineering and that my line hadn’t been transferred over, for some reason. He didn’t know why but he knew people who would but they would take between 1 and 6 business days to respond. By this time I couldn’t wait to see the back of Pipex.

I check my email and see that O2 have sent me notification that my line will be connected on 12/03/2008. I was slightly happier, but only slightly as that was almost a week away. So why didn’t Pipex know that the disconnect was to do with my transfer to O2? Why did the MAC code guy say what he said? If I had known that my internet connection being cut was due to the transfer I would have phoned O2 immediately and told them to start connecting me up. As it is I've had no internet connection for over a week.

I don’t think that any time is a good time to lose your broadband connection, maybe before a long holiday overseas. But this wasn’t the best time for me. I'd just put all my notes and to do lists into Backpack from 37Signals after reading the first couple of chapters of Getting Things Done by David Allen. So all my personal stuff and work stuff was all in one place. All completely inaccessible. I'd even signed up for a paid Backpack account to use the extra features, pages and storage. You just don’t realise how much you rely on it all until it’s not there. So far I have 2 A4 pages of notes, things I need to do and things I want to look up.

O2 have been great so far. Little text messages keeping me informed and saying when the router will arrive. The wireless router arrived today but I'm undecided about using that or sticking with my Belkin.

Tomorrow’s the big day. I've never been this excited about getting something back.

  • Posted on Tuesday, 11 March 2008
  • Tagged with moans, web

27 more posts tagged with 'web'…