Find Duplicate Files - dups.py

I have a lot of images imported at different times and from different sources, and I wanted a quick way to find duplicates. Not finding a satisfactory (read: free) solution (though I admittedly didn't do a very exhaustive search), I took this opportunity to learn Python and came up with dups.py. Note that the file displays within a frame, so you might have to view frame source to get to the actual code.

Without arguments, dups.py checks the current directory, recursively:

$ dups.py
Duplicates found:
./Data/2004/05_4/015_12A.jpg
./Data/2004/2004.09.29 Grandma/015_12A.jpg
Duplicates found:
./Data/2002/19/uvs021219-008.jpg
./Data/2006/01_2/uvs040430-006.jpg
  ...

This has been tested on the Mac OS X and cygwin, and should also work with Python for Windows.

There are lots of nerdy options, like filtering by file size and following symbolic links. Try dups.py -h to see them all:

usage: dups.py [options] [<file_or_directory> ...]

Find duplicate files in the given path(s).  Defaults to searching files recursively,
except for hidden files (beginning with "."), empty files, and symbolic links.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -v, --verbose         verbose

  Exclusion Options:
    -f, --flat          do not scan directories recursively
    -g n, --greater-than=n
                        only scan files of size greater than n bytes
    -l n, --less-than=n
                        only scan files of size less than n bytes

  Inclusion Options:
    -L, --follow-links  follow symbolic links (warning: beware of infinite
                        loops)
    -H, --hidden-files  include hidden files
    -z, --zero-files    include empty files

  Miscellaneous:
    -D, --delete        delete subsequent duplicates (files are scanned in
                        argument-list order)
    -c, --create-rel-links
                        replace subsequent duplicates with relative links
                        (non-Windows only)
    -C, --create-abs-links
                        same as "-c", but links are absolute
    -s, --special-hidden
                        changes meaning of "hidden files" (-H) depending on
                        platform:  cygwin - uses Windows file attributes
                        (warning: slow); win32 - files with names starting
                        with "." considered hidden

P.S. I hacked together a way to detect Windows hidden files from cygwin but it's ugly and slow.

4/6/08 update: I added the ability to delete duplicates (-D), and create relative (-c) or absolute (-C) symbolic links.

Hacking Lite - Evading Coffee Shop Banners

This is mostly a note to myself and not intended to express approval of the behavior described ;-)

Occasionally I like to bring my laptop to a nearby coffee shop to get some work done without all of the distractions of my apartment. My favorite place has been a Tanner's Coffee Company within walking distance of my place. It's a little noisy sometimes, and the food isn't the freshest, but the drinks are decent and I seem to get a lot done whenever I'm there.

Their wireless offering injects an ad banner at the top of every page. This alone would not be prohibitively annoying since adblock successfully strips the ads, leaving only the banner, but what does tend to dampen the customer experience is that it breaks some sites, Google Reader in particular. Because of this, I started to do a little tinkering...

I figured they didn't inject all internet traffic, since I'm able to ssh without problems. Maybe they detect requests to servers at port 80? I toyed with the idea of using a local proxy server, blah blah blah...

Turns out, they actually filter on the user agent field within HTTP requests! This means that if you're using Firefox or Safari (or, I imagine, Internet Explorer), the banner will be injected; Opera, however, is ad-free. This also means that simply changing the user agent field that your browser declares in its HTTP requests sets you (ad) free as well.

In Firefox there are a number of ways to do this: install a Firefox extension, or simply add a string value to about:config named:

general.useragent.override

with a value like

Opera/9.26 (Macintosh; Intel Mac OS X; U; en)

as described here. It's probably a good idea to stick with a realistic user agent string as opposed to something arbitrary, since websites like Gmail may switch to less functional versions if they don't recognize your browser.

A quick way to determine your browser's user agent is javascript:document.write(navigator.userAgent).

The service responsible for the ads at this particular Tanner's (I think they're all independently owned) seems to be a company named AnchorFree. Chances are, this technique could work for ad-injection schemes used by other wi-fi spots.

Done and done. Back to high-quality coffee shop web surfing!

5/1/08 update: Okay, I'm dumb. A much easier way to do this is to add the filter

*.anchorfree.*

in AdBlock Plus. This solves the problem much more elegantly and doesn't run into issues with sites not supporting your supposed user agent.

Navigate Gmail Using Your Apple Remote

When I first got my Mac, I was really impressed with Front Row and how easy it was to manage all of my media with an interface as sparse and simple as the Apple Remote. Yet, there was one thing that left me wanting -- "If only I could use it to check my email," I thought wistfully. "I'd never leave my couch!"

Finally, my ambition for laziness can now satisfied! Though lacks the shiny polish of Front Row, an application called iRed Lite allows mere mortals to customize the behavior of the Apple Remote. I've written what's called a "layer" in the iRed Lite parlance, which is basically a set of actions tailored for a specific application.

If this is something you're interested in trying out, a couple of caveats about my solution:

• It assumes you are running Firefox.
• It assumes you are always logged into Gmail (by checking "Remember me on this computer" at login).
• So far it seems that iRed Lite cannot save an entire layer at once; each action must be saved as a separate script. I've created an archive called iRedLite_Gmail.zip consisting of actions which I map in the following manner:

Open - browse to Gmail	double-click play
Open conversation (o)	play
Back to conversation list (u)	left
Star conversation (s)	hold play
Previous conversation (k)	up
Next conversation (j)	down
Previous message (p)	right
Scroll down	double-click down
Scroll up	double-click up
Increase text size (cmd-=)	double-click right
Decrease text size (cmd--)	double-click left

Of course you can easily remap them any way you wish. Beware that iRed Lite tends to crash here and there, though -- save early, save often.

To recap, first download and install iRed Lite. Then download and unzip iRedLite_Gmail.zip, create a new layer and import the actions defined in each file in the archive.

3/15/08 update: Thanks everyone for your comments! To summarize:

• My actions have been grouped into one file here.
• You need to enable keyboard shortcuts in your Gmail Settings for this to work.
• This doesn't play well with Firefox Beta 3; I'll do what I can to tinker with this:
  • You'll need to change the "increase font size" shortcut to send "cmd-+" instead of "cmd-=".
  • "Open Gmail" doesn't work.

I will work on a Google Reader version at some point. I will incorporate everyone's suggestions and post updates here.

From Your Side

Here are the lyrics to a song I wrote back in my less heathenistic days. I have an entry on my to-do list to record this some day but I haven't been very motivated to get it done.

From Your Side

There was a day
I thought I knew just what I wanted
But my heart had gotten in the way

There was no use
I was sullen as a stone
I didn't have a thing to say

I wish that you could carve my heart
So that I could have another start

But we all have our demons
And I know you have your reasons
I just wish I could see the other side

What does it really matter
You tell me there's an answer
But I can't help thinking that there must be more

I know it's not about me
But sometimes it's not so easy
I know the picture's bigger from your side
From your side

There is a way
And though I know which way to go
It doesn't wash the pain away

So now today
There is something I've got to do
I'm going to give it all away

I wish that I could see your eyes
'Cause then I think I'd realize

When the hunger is so strong
And I know I can't go on
You're always there to tell me it's all right

Could it be that I'm too late
But you're so willing to wait
There are better things than this world, in the next

I know it's not about me
But sometimes it's not so easy
I know the picture's bigger from your side
From your side
From your side

Dear John Letter

Here is my email to John Oakes. It's a bit less derisive than my earlier post; I just listened to the lecture and his approach is quite respectful and less self-assured than I was led to believe by just reading the notes. Nevertheless the content is the same and so I feel my criticism is warranted.

I just listened to your lecture on The Problem of Pain and Suffering and had a couple of comments.

Before I elaborate, let me mention that I applaud your respectful treatment of the difficulty that the question presents to the Christian. I also appreciate the fact that you address the sources of suffering separately, namely other people, and natural causes. It is with your explanation for the latter that I take issue, however.

First of all, I do not believe it is up to "us" — whether unbelievers, or doubters, or objects of God's creation — to propose a universe which does not include plate tectonics, before we can rightfully criticize the concept of an omnipotent God who created a world where suffering comes at the hand of that same world. It is the bible that makes the claim that such a God exists; thus the burden of proof lies with the bible — or at least the theist who claims to believe it — to sufficiently explain this assertion.

Second, would you have us believe that God spoke this world into existence, can change the nature of physics at will to enable a man to walk on water, and yet cannot save people from earthquakes caused by the plates of the earth shifting because they are necessary for life? Or, what of the virgin birth? And, is it not Jesus' unique ability to nullify the natural order of life and death, the very reason that we should believe he is from God? And yet, this same God must now submit to the very same laws of physics he so remarkably violated before? This approach seems very inconsistent.

You went on to include other phenomena such as hurricanes, tornadoes, floods, and even bacteria. Unremarkably, then, you have made God subservient to exactly the circumstances of the natural world in which we find ourselves. If this is the case, then wouldn't it easier to assume that God does not exist, or at least does not care? At least this would relieve us of the aching burden of searching to find some purpose for senseless suffering.

Consider this: are there plate tectonics in heaven as well? If heaven is some different kind of existence, why not just start with that existence? This solves the problem of envisioning an environment that does not include plate tectonics. God could still accomplish his goal of "soul-making," or whatever other justification one might have for suffering at the hand of other humans with free will, without adding the additional burden of suffering from natural causes. Thus I respectfully find your explanation lacking.

Another issue you may wish to address in your talk is the problem of animal suffering — that is, if animals do not have souls and do not have the chance to go to heaven, their suffering has no meaningful explanation. As of yet I have not heard much argument from the apologetic side on this issue (I have not yet listened to the rest of the conference lectures so I apologize if this is addressed elsewhere).

Thanks for your time.

The Hills are Alive! Run!

On June 30^th I joined one crazy crew of fans at the Sound of Music Sing-A-Long at the Hollywood Bowl. It's kind of like the Rocky Horror Picture Show, but for kids. And strange fanatical adults.

These photos were shamelessly stolen from Claire. Of course I only stole the, uh, relevant ones.

Claire kept snapping pictures so I decided to start pointing out the object of interest in each one, in case it wasn't obvious. This is the only example I stole from her site though.

We pretty much baked for the first hour or so before the sun dipped past the trees, during the pre-game entertainment.

The pre-game was emceed by Melissa Peterman, who was actually quite entertaining. So much so I am almost tempted to check out an episode of the show of Reba where she apparently is part of the cast. Almost.

The real endurance part of this mad race was the endless costume parade. Of course it's cute in the beginning but wow, two hours? Yikes.

One of the winners of the costume contest is pictured here. They're the curtains that Maria uses to make play clothes for the kids.

One of the great scenes in the movie, complete with subtitles by which to sing-a-long.

This (third from the left) is the actress who played Liesel in the film.

Heh, I'm pretty wiped out at this point. All in all, lots of fun! Thanks to Steve and Sophia for putting it all together.