Russell's Blog

I finally got fed up with the pathetic official Android release from T-Mobile, and rooted my G1 and installed the CyanogenMod firmware. Cyanogen feels about twice as responsive as Cupcake! It's like a whole new device.

Also, the tethering app is awesome. It turns your G1 into a WiFi base station and routes traffic from WiFi to 3G. Since I'm still waiting for broadband at my new apartment, it's a lifesaver.

I suppose tethering (and rooting the phone) technically violates T-Mobile's TOS, but I'm convinced that T-Mobile will allow both sooner or later. It's just too awsome, and it would help them sell more contracts.

It's kind of difficult to abuse tethering anyway; it sucks down the battery very quickly, and the latency is significant. It's the sort of thing you'd only use in a pinch. Those happen to be the situations where a little benevolence or selfishness from a big company can shape a customer's opinion forever. T-Mobile seems to be more sensitive to that kind of thing than the other networks. I know they've got their reasons for banning tethering apps, but I think they could be convinced to change their minds. (You can download various petitions from the Android Marketplace.)

Openness is where Google and T-Mobile could really go after the unwholesome, anticompetitive and un-American AT&T/iPhone alliance. The open nature of Android is a step in the right direction, but T-Mobile needs to get its legal department on the Open Access bandwagon if it wants to press the advantage.

After all, if some random people on the internet can roll better firmware for the G1 than their in-house developers, isn't it a strategic business advantage to let them?

I've been trying and failimg to get to sleep. The average interval of cat bombardment, and the mean momentum of the incident cat field has been unusually high. It took me a while to realize that this was related to the weather; it is lightly sprinkling. My cats have never seen rain.

That's a sobering thought in two ways. On one hand, they are old enough to have very well developed personalities, and yet are younger than the Obama administration by several months. On the other hand, it's been that long since it rained.

This summer has turned out a bit like a chocolate bar in the sun; everything started out nice and neat, but everything ended up squished into the bottom of the wrapper.

Right now, I'm working with Andrey Kislyuk on our little piece of the DARPA FunBio project. We're in the middle of a two-week code sprint, so I'll save that for a later post.

I also moved to a new apartment, and that didn't go nearly as smoothly as it could have. The guy we subleased from was in the process of buying a house, and the loan underwriter decided to yank back the money after he'd closed escrow (or was in escrow, or something). Evidently they wanted a sworn affidavit from the gardener that he was contracted to take care of the grounds. Anyway, the upshot was that instead of a nice leisurely move, he got stuck in the apartment for three weeks longer than he expected, and I was homeless for a week. Fortunately, one of the staff scientists in our lab was generous enough to let me stay at his apartment. Neil and Buzz got to learn about stairs, which they evidently adore.

Over Labor Day weekend, I went with Srijak and some of his friends from San Diego on a day hike at Lassen Volcanic National Park. I've always loved California, but it's nice to be reminded from time to time exactly why I love this place so much.

Because it's awesome.

If you should happen to be filling out a longish web form (say, for example, a blog post) using the Android browser, and someone calls you, the browser will dump all your windows and obliterate the POSTDATA. Gee, thanks. That's exactly what I wanted.

Come on people.

I want announce a little project I put together over the weekend. For want of a better name, I'll call it the Davis Crash Map. Basically, I analyzed the accident report spreadsheets from the City of Davis Public Works Department, and made an overlay for Google Maps to visualize the data. The spreadsheets are a bit difficult to analyze, so I'm leaving out the reports that aren't clear to me (about 15% of the reports). The reports that gave me some trouble seem to be randomly distributed over the city, so the overlay should still give an unbiased picture of what is happening.

In particular, this is map is intended to examine bicycle accidents. I hope people will look at this map, and think about how they behave on the roads, weather on foot, on a bicycle, or in a car. How you behave on the road has direct, and sometimes dire, consequences for you and for other people.

However, there is more to this than behavior. This is also a design question. Roads are not natural features. They are designed and built by people for use by people. As with anything that is made by humans, there are good designs and bad designs. These designs have a real impact on peoples' lives. In the case of streets, the impact on your life can be very literal, as this map shows.

Even good designs can always be improved. Davis is a pretty safe town in which to walk, bicycle and drive. But if you study this map, and think about it as you go about the town, it's also clear that things could be better.

I'm not a traffic engineer, or a civil engineer, or a city planner. I claim no expertise in those areas. I'll leave it to other people to make specific suggestions. However, I think it is important for the users of streets -- pretty much everybody -- to think about what kind of streets they want. This map should help give you a better idea of what kind of streets we actually have.

For some reason, people seem to get very emotional about traffic. I grew up in Los Angeles, home of the nation's worst traffic jams. Perhaps this is to make up for our lack of a professional football franchise. Passions about transportation, especially mundane things like parking spaces and HOV lanes, get people really worked up. Los Angeles is also famous for road rage, and nowhere is it in greater evidence than in the corridors of City Hall. Public meetings on traffic can make I-405 look like afternoon tea. In fact, thousands of people from all over the world tune into the internet broadcast of the Santa Monica city council meetings to listen to Californians scream at each other over the exact position of little blobs of paint on little strips of asphalt.

What the conversation needs, I think, is some perspective. Data can help provide that perspective, especially if it can be represented in a way that is easy to understand. Maps are good at that.

If you will indulge me, I'd like to share my perspective on this data. Each marker represents a traumatic event for someone. Under some of those markers, a life came to a sudden, violent end. I'd like to share a picture of what kind of event a marker on this map represents. You won't find a marker for this event because it happened in Norman, Oklahoma, a college town that is a lot like Davis.

Anna and me

In October of 2007, my little sister was riding her bicycle near her house. A lady in a Mercedes made a lazy left turn, and crossed onto the wrong side of the road. She hit Anna head-on. Anna went up and over the hood of the car, and face-planted on the windshield, breaking her nose and her front teeth. The lady slammed on the breaks, and Anna then went flying off the car and slammed her head on the pavement. That much is clear from where my mother photographed the tire marks, the blood stains, and scattered teeth.

Who designed this street, anyway?

The sequence of events afterward are a little unclear, since Anna does not remember anything from that day, or for several days before and after the accident. The police report includes several details that are impossible or don't make any sense; for example, the officer thought she was coming out of a driveway onto the street, but the driveway did not belong to anyone she knew, and was paved in gravel (extremely annoying to bicycle on). The report also places the accident on the wrong side of the street, which was obvious enough based on the tire marks and blood. Based on what her friends say she was doing -- biking from her house to a friend's house -- she would have just been pedaling along the side of the road. The details of what happened are somewhat unclear, other than the evidence left on the road and gouged onto my sister's face.

After hitting the pavement, she evidently got up and staggered around for a bit, and then collapsed. She stopped breathing, and officer on the scene couldn't find a pulse, and assumed that she was dead. This was the reason given for not immediately summoning an ambulance.

Then she suddenly revived and started mumbling. The lady who ran her down went into screaming hysterics, and had to be restrained (or evacuated, or something). It was only then that an ambulance was called. From the report, it appears that paramedics and police spent a good deal of time tending to the driver of the car, who was having an anxiety attack, instead of Anna, who was bleeding from massive head trauma.

Anna then spent the next several days in the hospital. My mother got on the next flight to stay with her. For the next several days, Anna went through long and short memory lapses and dizzy spells of various lengths. When I spoke to her on the phone over the next several days, she also had some kind of aphasia, which was very jarring to me because she is normally a very articulate person. And then there was the puking. Brain injuries often come with a heavy dose of overpowering nausea. She was on anti-nausea drugs for a long time after the accident.

It took a long time for he to start feeling "normal" again. Almost two years later, she's still not sure she feels completely normal. Fortunately, thanks to some really great work by her surgeons, she looks normal. Needless to say, she is both very lucky and very tough.

Anna's bicycle. The police kept it as evidence, but allowed my mother to photograph it.

You could say that I have a personal stake in this, and I will not claim to be unbiased. Many people who argue against safety measures that would slow traffic argue their case on the basis of personal responsibility. We are each responsible for our actions, they argue, and if you do something stupid, you are responsible for the consequences. Why should people who don't do stupid things be inconvenienced?

I agree completely. However, if one casts any real issue into the frame of personal responsibility, then things are rarely so simple. Everyone who could act in a situation has responsibilities, even if they are not they are directly involved. When you have the power to prevent something bad from happening, and you choose not to act, then some of the responsibility falls on you. Every unfortunate, stupid thing that happens involves a cast of thousands of silent, but not blameless, bystanders.

We have a responsibility to at least attempt to protect people regardless of what they are doing -- even if it is stupid. This is especially true when it comes to the things we build. We shouldn't, if we can possibly avoid it, build things that injure and kill people. If we can think of ways to make something we build less dangerous, we ought to give it a try.

Anna and Earnie, about a year after the accident.

My little sister was stupid not to wear a helmet that day. The lady in the car was stupid not to have been on the lookout for cyclists. But neither of them deserved what happened. Each of them is obviously bears some measure of responsiblity (and I have my own opinions on how those measures are apportioned), but the city of Norman also is also responsible. The city didn't even bother to paint a line down the middle of the road; what was the driver supposed to be on the wrong side of?

Yes, this is about personal responsibility. We, the public, build the roads. We are responsible for the markers on this map, and all the terror, trauma and tragedy they represent. Let's try to do better.

I've been tinkering with a little data visualization applet for looking at bicycle crash data in Davis, and I thought this map might be interesting to people. This is a image was generated with Google Maps and a heatmap overlay generated with a gheat tile server.

This is for 168 bicycle accidents that happened between 2004 and 2006. I have a lot more data, but 95% of the work in this little project involves parsing and renormalizing it. Evidently, police reports are not written with data processing in mind! I suppose that makes perfect sense. An officer at the scene of an accident probably has things on her mind besides generating a nice, easy to parse data point for future analysis. The priority seems to be completeness, rather than consistency. My parsing code, for example, has to be able to correctly detect and calculate distances measured in units of "feeet".

I'll release the applet here once I make an interface for it (and get the rest of the data imported). Stay tuned.

I have a bad habit. I often open my IMAP mailbox directly on the mail host with mutt. This inevitably causes occasional complicated messes when I butterfinger something. For example, a few days ago I accidentally moved all of the read messages to a separate mailbox. I merged the message back in, but not before my desktop, which was patiently monitoring my IMAP mailbox on the other side of town, decided to synchronize a few thousand messages with the IMAP server. This resulted in lots and lots of duplicate messages.

The trouble was, the duplicate messages had different X-IDs so, their MD5 hashes would be different. After fiddling around with formail for a few minutes, I got impatient and banged out this fun little Python hack :

import email, imaplib, getpass

M = imaplib.IMAP4_SSL( '**********' )

typ, data = M.login( getpass.getuser(), getpass.getpass() )
if typ != 'OK' :
    raise Exception, 'Login failed.'

typ, data = M.select()
if typ != 'OK' :
    raise Exception, 'Selection failed.'

typ, data = M.search( None, 'ALL' )
if typ != 'OK' :
    raise Exception, 'Could not get message IDs.'

id_list = data[0].split()
mids = []
for id in id_list :
    typ, data = M.fetch( id, '(RFC822)' )
    if typ != 'OK' :
        raise Exception, 'Could not fetch message ' + id
    mail = email.message_from_string( data[0][1] )
    mID = mail.get( 'message-id' )
    print mID
    mids.append( (mID, id) )

mids.sort()

dupes = []
for i in range(len(mids)) :
    if m[i] == m[i+1] :
        dupes.append( m[i+1] )

print 'Found ' + len(dupes) + ' duplicate messages.'

for m in dupes :
    typ, data = M.store( m[1], "+FLAGS", '(\\Deleted)')

print 'Marked ' + len(dupes) + ' for deletion.'

typ, data = M.expunge()

print 'Expunged ' + len(data.split()) + ' messages.'

Duplicates begone!

It's a little annoying that imaplib doesn't have a friendly wrapper function for marking messages for deletion, but M.store( m[1], "+FLAGS", '(\\Deleted)') does the job just fine.

For the last couple of years, I've been running Typo, a Ruby on Rails blogging tool. It was nice, but there were a few persistent problems I encountered :

• It was sloooooow. Nothing I did seemed to get it to run faster, even with carefully tuned caching.
• It was unstable. Typo would run happily for months, and then mysteriously explode. This usually happened while I was traveling, or busy with something more important.
• It was difficult to fix. Usually, when Typo would come down, it took a few days of research and pestering people to figure out why.
• The database migrations between versions were awful. You'll notice that the first year of posts don't have any tags. They were deleted by a bad migration. I have backups, but merging them back in is nightmarish.

A lot of the problems I experienced are with older versions of Typo. They've definitely gotten better over the years. But there is one big reason I'm abandoning Typo; I don't want to code in Ruby. I do most of my coding in python. I like python. I seems silly not to use the knowledge I have from doing computational physics in python.

I have used Blogmaker for most of the main elements on my site, but with a fair bit of hacking to make it do more of what I want. I also wrote a Typo-to-Django import utility, if anyone is interested. The URLs are slightly different, so I'm going to watch the 404s for a few days.

Languages like Python are great for tasks that require manipulating complicated data. The main complaint about such languages is that they aren't as fast as compiled languages. For scientific computing (and a lot of other things), that's a show-stopper. However...

from weave import inline
a = 25
code = \
	"""
	int i = a;
	while( i > 1 ) {
		printf("a number: %d\\n",i);
		i = i / 2;
	}
	return_val = i;
	"""
inline(code, ['a'])
===output===

a number: 25
a number: 12
a number: 6
a number: 3
1

My mind just spins thinking of all the horrible, horrible things I can do with that.