skwpspace it’s pronounced ’scoop’. yan pritzker’s home on the web

Twitter opens the floodgates of FUD

TechCrunch is reporting on rumors that twitter is leaving Ruby on Rails. Of course the comment threads are covered by a heated debate by people either bashing RoR, suggesting their favorite language and platform as the ‘only possible solution’, or both. Nevermind that Friends For Sale, with 630k active daily users scales just fine on RoR. Let’s ignore that scribd and yellowpages.com use RoR and most developers would only dream of having the traffic these apps have.

Of course, only Enterprise Java scales, and we know this because the word Enterprise is right there in the name. Of course, the only platform you should consider is .NET because it has built in caching and hundreds of thousands of college graduates know it. Of course, you need to use PHP because Facebook is written in PHP and Facebook scales. The fallacy of these arguments should be painfully obvious, yet people are shouting these things as if they were gospel truth.

I’d like to offer here, the scalability manifesto. Please repeat these items aloud to yourself every time you want to say something about language X or framework Y not scaling.

The Scalability Manifesto

1. Scalability means the ability to handle increasing load by increasing resources.
2. Scalability does not mean being ‘fast’.
3. Choice of language does not guarantee scalability.
4. Hardware costs decrease exponentially, developer salaries do not.
5. Only you can make your system scale.

Trading performance for development time is generally not a good idea. It’s likely that by the time your application reaches serious traffic levels (if ever), hardware will cost half of what it did when you started. Now does spending twice as much dev time with a lower level language seem justified? Now this is not always the case, but my bet is if you do the math and realize your team of 4 Ruby hackers is doing what a team of 10 Java programmers used to do, you might conclude that linear performance gains due to language choice are not as relevant as hiring smart people who can get work done quickly, and design scalable systems.

There’s no such thing as a free lunch, and that includes scalability.

Ponoko brings product design to the masses

The printing press, the photography studio, and the movie set are already in the hands of the people thanks to products like wordpress, flickr, and youtube. Now Ponoko is looking to revolutionize product manufacturing by providing a direct channel for users to create designs, have them manufactured to order, and distributed via an online storefront with zero upfront costs. You pay nothing unless your product gets sold. A variety of materials including acrylic and MDF are available in many colors. Designers simply upload their 2d or 3d files and choose materials.

Essentially, designers become able to produce any of their ideas without having to look for a factory to do limited production runs, which can be extremely expensive if at all possible. Democratization of the means of production has already created many interesting new ‘long tail’ genres (the overphotoshopped but interesting flickr photo, the silly viral video, etc). I wonder what kind of interesting long tail products will come out of the ability for anyone to be a product manufacturer. There already quite a few interesting ones in the ponoko showroom.

Even cooler, some designs can be made open, which means other users can download and modify your design and improve or tweak it to create their own products. Open source product design, how about that.

More git tips and tricks

Here are a couple more fun git commands and configuration tricks.

Make some aliases because typing ‘checkout’ is a pain. In ~/.gitconfig:
Note the quicklog command below uses a custom log format which shows you a short hash, a relative date (”4 hours ago”) and the commit message. I created it by reading the git log formatting manpage.

[alias]
  ci = commit
  co = checkout
  b = branch
  nb = checkout -b
  stat = status
  quicklog = log –pretty=format:\”%h %cr %cn %Cgreen%s%Creset\”
  changes = log –pretty=oneline
  svnup = svn rebase
  svnpush = svn dcommit

I really love the ‘nb’ alias, which I use to mean ‘new branch’. It lets me very quickly create and switch to a new branch “git nb fixing_bug_123″.

Display the current branch in your bash prompt. Modified from here.

export MANPATH=/usr/local/git/man:$MANPATH
 function parse_git_branch {
   git branch 2> /dev/null | \
	sed -e '/^[^*]/d’ -e ’s/* \(.*\)/(\1) /’
 }
export PS1=’\u@\H:\w $(parse_git_branch)$ ‘

Get a nice list of change summaries, one line per change

git log --pretty=oneline --since="2 weeks ago" --until="yesterday"

See what changed from one commit set to another (these can be branch names)

git log my-branch..master

See which files changed in the log

git log --name-status

Browse your local repo with a web interface

git instaweb --httpd=webrick

Want to reorder previous commits, change log messages, and generally wreak havoc?
The following command opens up an interactive editor with the last 10 commits.

git rebase -i HEAD~10

bluehost.com ignores customer complaints on database downtime

I gave bluehost more than two months to fix their issues, or even simply notify their customers of ongoing problems and make some sort of gesture to make up for the downtime. They have done neither, while the server that hosts this blog (box 12 at bluehost) continues to have constant database downtime.

I’ve included a one liner history of my interaction with them. My responses are paraphrased, while theirs are direct quotes from their emails. Note that they usually respond hours later when the site has restored itself due to their reaper scripts constantly restarting the database.

• February 13 Yan: Database problem.
• February 13 Bluehost: The processes on your server have been shut down. The site is showing up and running.
• At this point I call them, have a chat, they say they have had ongoing problems and they will be fixed. I ask them to notify customers. They do not.
• Mar 9 Bluehost: Currently MySQL on your server (box12) is overloaded.
• Mar 10 Bluehost: It looks like you are exceeding your own mysql active connection limit which is 30
• Mar 10 Yan: How can I exceed 30 connections with a blog that gets low traffic and runs 5 php processes? Database is down again.
• Mar 10 Bluehost: At this time there are no connections to terminate.
• Mar 17 Yan: Down again.
• Mar 17 Bluehost: Checking the sites both show up an running. At this time there is no issue at this time.
• Database goes down 10+ times between March and April. I just wait, hoping they will fix it.
• Apr 17 Yan: Down again.
• Apr 17 Bluehost: I have checked your databases, and do not see the connections maxed out.
• After several angry emails, my ticket is escalated to the next level of support. The person seems more intelligent than the previous support staff who clearly have no ability to investigate past events.
• Apr 18 Yan: Why do I see scripts that constantly restart mysql?
• Apr 18 Bluehost: When there are problems with mysql the server will try to restart it.
• Apr 18 Yan: Brilliant answer.
• Apr 25 Bluehost: I can see that Mysql has been having some problems on your server. (Ya think!?)
• Apr 25 Yan: Who is eating my connections? I asked before and no one answered how 5 php processes are eating 30 connections.
• Apr 25 Bluehost: [...]Not necessarily from your account[...] To be honest some boxes seem to have more problems then others and box 12 happens to be one of those boxes.
• Apr 25 Yan: How about telling your customers that box 12 has some capacity issues and moving them off?
• Apr 28 Bluehost: I have been keeping a close eye on box12 it hasn’t been having any issues the last few days. There is no capacity problems.
• Apr 28 Yan: Down again.
• Apr 30 Bluehost: no response….
• I call them up. The support person verifies that there have been ongoing issues on box 12. I ask why the customers have not been notified. She says because they prefer to spend their energy on fixing the problem and probably most customers don’t use the database. I say, fine, but it’s been 2 months. She says she doesn’t have a good answer. I inform her that their failure to be proactive is resulting in this blogpost.

Well, that’s the story. Draw your own conclusions. Either there are some serious miscommunication issues inside this company, or the support staff have been lying to me outright. In either case, it’s not a company I wish to continue to do business with. I’m looking for a new hosting provider to move away from Bluehost.

They’ve moved me to a different server now in hopes that this new home will have less downtime…however as of today they have still to my knowledge not contacted anyone else on box 12 to tell them about ongoing problems.

Cordarounds - sweet pants, sweeter customer service

I recently had a great experience with Cordarounds, an independent clothing company run by Chris Lindland. I stumbled upon the website I believe through del.icio.us, and found it to be a fun, quirky place that doesn’t take itself too seriously.

I ordered some pants and a blazer from them, and while I really dug the pants, the blazer was a bit large, so I had to return it. It took them some time to get to my return but Chris himself contacted me by email, and with a couple instantaneous email exchanges and a phone call, my return was processed very quickly. With inbox bankruptcy becoming a fad, it’s nice to see some people still realize that people love to get instant replies to their emails. Instant reply = instant satisfaction, no matter the content :-) Thanks for the quick service, Chris!

Does Mysql really need occasional restarts?

You may have seen my blog occasionally go down due to database problems. I have been constantly on email and phone with bluehost support to try to figure this out. The responses I’ve gotten always seem completely ignorant of bigger issues. Ranging from “we didn’t see a problem” to “we killed the threads so everything should be fine now”. The last time it happened, I was online at the time and literally saw some reaper scripts fire up and nuke the Mysql server and cause it to restart. So I decided to investigate why this was happening with support.

After about a round of ten emails, I got escalated to level 3 support and got the best “you’re in a helicopter” response ever: I asked why they have scripts that run that kill and restart mysql after I caught this happening during a time when my blog was down. The answer: “When there are problems with mysql the server will try to restart it…”. Uh, really?

Now maybe I’m crazy here, but can I get a show of hands of people who have such problems with their Mysql that they have to regularly restart it? In my experience, it’s possible to run a properly configured database instance with nearly infinite uptime. So, are they royally screwing something up? Or is this really just a downside of running a database for tons of clients who may be doing bad things to it?

Update: bluehost has admitted the problem is not from my account. They have a connection limit of 20 connections for Mysql for all accounts on the server. That it took me ten emails to have them admit they’re overselling capacity is downright unacceptable. And I’ve been fighting this battle with them for more than two months now. I asked them time and time again to admit they’re overselling and to fix the problem by reducing load on the server, but no one would so much as even answer my question, every time implying that it was my php scripts (a standard wordpress installation, using caching, and on low traffic with less than 5 php fastcgi instances) causing the problem. Now they’re offering to migrate me to another server, clearly again band-aiding the problem instead of addressing the core of the problem.

From bluehost support:

“When you get the error that mysql is out of connections that means mysql connections on your server have backed up to the point to where the connections are maxed. Not necessarily from your account, but the server as a whole. We don’t think a restart is the answer to the problem, however often times stopping mysql, running a repair to fix any corrupt tables then restarting will often times clear out the cause of the problem.”

We cannot receive your email at this time

I submitted an inquiry to B&H Photo, an online Photo Store that has a cute habit of ‘closing’. Evidently these people didn’t get the message that online stores don’t have hours or off days. But here’s the kicker, I emailed their customer service for an inquiry, and this is the response:


We regret that we cannot receive your e-mail at this time. We will reopen on Monday, April 28th at 9:00am. Please Re-submit your inquiry after that date.


Newsflash guys: this is the Internet. Just because your store is ‘closed’ doesn’t mean you can’t accept email. Asking your customers to re-submit an inquiry is plain silly. How about a nice friendly “We’ve received your email and we will get to it as soon as we get back.” What is the world coming to?

Elastic Server now with Passenger (mod_rails)

We’ve added Passenger support to our Rails 2.0 Elastic Server Portal, so that you can now create both Mongrel and Passenger servers and compare them. If you’re happy with what you’ve built, create it in EC2 format and deploy it to the cloud! Read more at the Elastic Server Blog

Useful git commands and quirks

Things I’ve learned from my first git experiences. First of all, git is inconsistent as hell. Every command has its own quirks and syntax, so I’m attempting to catalog some of them here. We’ve been using git svn at Planypus as a way to maintain local developer branches and still push to our svn as a central place which then triggers our integration server rebuild, cruise control, and so on.

First, make it look nice

$ git config --global color.diff auto
$ git config --global color.status auto
$ git config --global color.branch auto
$ git config --global color.interactive auto

via tpope

Onwards to the commands…

If you’re using git svn and you want to see changes that you have comitted locally but have not yet dcomitted

git svn dcommit -n

This might give you output that looks like this
diff-tree e784cfa... e784cfa...

BUT, if you want to actually see the changes, what you want is apparently the ‘patch’ output:

git diff-tree -p e784cfa... e784cfa...

If you have uncomitted changes, it’s easy to see them

git diff

BUT, if you’ve already created your commit set (git add) but not comitted, you how have to use

git diff --cached

If you’ve accidentally added some files to your commit set and don’t want them in there

git rm --cached [files]

To undo local changes (revert in the svn sense)

git checkout [files]

BUT, to reset all changes that you’ve had

git reset --hard

If you’re working on something and you decide you want to scrap it or work on it later

git stash

There are lots of fun things you can do with stashes but they’ve been covered on other blogs.

on merging…

If you’ve done a git merge and have some conflicts, launch git mergetool which will take you through the conflicted files one by one using your default merging tool (FileMerge, usually, on OSX) and let you merge them. However be careful. Either because of a misconfig on my end or some other quirk, FileMerge shows the file I pulled from svn as “LOCAL” while my file with changes is “REMOTE” which is counterintuitive to me and caused me to fubar a merge the first time I tried.

Take note that after the merge, you will have files ending with .orig left on your system so you’ll have to manually delete them, unless you have the config mergetool..keepBackup set, but I haven’t tried this yet. Also, after the merge I was getting error messages about having a .git/MERGE_HEAD file, which I couldn’t find enough docs online for, so I just deleted it..and hopefully that did not fubar me.

Well that’s all for now. More might come as I learn to stop worrying and love the git.

P.P.S Many of these commands are not necessary if you use The Git Textmate Bundle and GitNub which integrate together and work very well!

The security of working at a startup

I recently had an interesting conversation with my grandfather, who was concerned about the economic downturn (recession?) and the impact it may have on my having a job. As I tried to explain to him why I choose to work in startups over big companies, I came to realize that I feel there’s a certain job security in working in a startup, perhaps even better than that of a big company.

In a startup you know when your money is going to run out. In a startup, you are responsible for the existence of own job position. By this, I mean that in a team of ten, you know that slacking off is not an option. You know that the work you do will directly result in business for your company, investment, and hence the ability of the company to pay you. Your own contribution carries a high proportional weight, which means you have a very real effect on whether your position is still available tomorrow.

A big company is an entity with a life of its own. The average employee does not have a sense of their own impact on company growth or business. Hiring and firing decisions are made on a level far enough removed from the average worker that they feel such decisions are arbitrary and unjustified. Employees talk in hushed whispers about looming layoffs without understanding why they are necessary. After all, they are just doing their jobs!

In a bigger company, layoffs due to a downturn in business or the economy are not only possible but probable. In a startup, they are unaffordable; A team of ten that loses one person loses 10% of their workforce, and faces a huge challenge in bringing a replacement up to speed in such a fast paced environment. A company of thousands can lay off handfuls of people here and there in a heartbeat in order to course-correct.

So - which is more secure? Sure you can look at a big blue chip company and say - they haven’t laid anyone off in ten years. But do you know for a fact that they won’t lay you off tomorrow because of factors you can’t even control? Think about that when deciding on your next job, and choose to work in a startup, where you are responsible for your own success (and failure).