More Github failures

Output from my trusty svn_up script, which also pulls git:

git detected in arkx
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in arkx, retry manually?
 
git detected in couchrest
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in couchrest, retry manually?
 
git detected in globalite
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in globalite, retry manually?
 
git detected in masquerade
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in masquerade, retry manually?
 
git detected in rack
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in rack, retry manually?
 
git detected in relaxdb
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in relaxdb, retry manually?
 
git detected in rest-client
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in rest-client, retry manually?
 
git detected in thin
executing: git pull
github.com[0: 65.74.177.129]: errno=Connection refused
fatal: unable to connect a socket (Connection refused)
!!! command ‘git pull’ failed in thin, retry manually?

Does anyone else think that it is not really acceptable for a large number of important source repositories to just go down at the drop of a hat?

UPDATE: from github.wordpress.com:

github 7:46 pm on August 23, 2008 | 0 | # |
We’ve taken the site down for approximately 3 hours while we’re upgrading our servers.

THREE HOURS??!!!

Tags: ,

14 Responses to “More Github failures”

  1. Wincent Colaiuta Says:

    Not too crash hot for a paid commercial service. If I were a customer I wouldn’t be too happy (luckily, I am quite sure that I never will be a customer).

  2. Says:

    We had a banner on every page forewarning people about the downtime. We hated taking the site down for that long more than anyone, but it was a necessarily evil to restructure our hardware.

  3. Sho Says:

    I hear you, but you should have arranged some sort of backup, even if it was read-only. I am assuming you are talking about new storage, since that’s the only thing I could imagine taking 3 hours. You could have kept the old one online as read-only, while prepping the new. Well, I’m just guessing anyway so whatever.

    I can handle the web site going down, but the native SCM access was a bit much. And I didn’t see any of those banners, it was a total surprise to me. Perhaps you should consider email alerts for extended downtimes in the future. Relying on users to notice ad hoc warnings on the web site isn’t really reliable.

    Ah well, not like i’ve given you guys any money ;-)

    PS it’s still SLOOOOOWWWWWW

  4. Wincent Colaiuta Says:

    Good idea the one of a read-only backup. In any case with Git there should never really be any excuse for a single point of failure taking down a project; it’s distributed after all. If the “Foo” project wants to host its “official” repo on GitHub then there’s nothing stopping it from having redundant clones on repo.or.cz, gitorious.org (both free, there are others), or “foo.org” itself. You can even set this up using hooks to have the push out to the other repos occur automatically.

  5. Sho Says:

    Wincent: yes you can, but not many people do this as far as I know.

    I think currently, since git is fairly new (at least in terms of its popular acceptance) people are still treating it basically like a better-working subversion, and its true capabilities haven’t really caught on yet.

    I admit I was surprised at first to see the eagerness with which Github was adopted, but I now admit there are benefits in putting all those projects in once place. I believe the advent of Github has had a positive effect on open source collaboration, at least in the Ruby sphere. However, in time, I’d like to see some diversification, and would like to see Github as just another peer in a much larger, properly distributed, system.

  6. Wincent Colaiuta Says:

    Agreed, most people use Git with a “Subversion-like” mentality and I think that’s likely to continue pretty much forever. That’s because even if you have a bunch of repos distributed across the globe people still like to have “the” official, central repo where they can go to get the “official” version of something. This is only natural, of course; what’s the point of even giving our projects names otherwise? You give something a name to assign an identity to it; and identity means “single” or “one”.

    So I don’t really see any problem with the way people are using Git; it’s the way I use it and will continue to do so. The fact that it’s distributed just happens to be a bonus that allows you to work quickly and offline, conveniently replicate a repo, track an upstream project, maintain a fullblown fork, or just track a set of custom changes that you like to apply on top of another project etc.

    I too was surprised to see GitHub embraced so wholeheartedly and so suddenly by the Rails/Ruby community. I was principally puzzled because there were plenty of free, open-source alternatives out there and GitHub is both commercial and closed source. I was also puzzled because publishing a Git repo is incredibly easy and there really is no need to pay (or ask) for someone else to do it for you. Basically every project out there has a web presence already and that means that basically every project could self-publish their Git repo if they wanted. It really is very easy.

    So it’s a bit of a mystery why people got so excited about GitHub. I know they emphasize the social aspect of it. Even without GitHub, forking someone else’s project is as easy as running “git clone”; but the difference with GitHub is that these aren’t private forks. Normally if you want your private fork to be publicly visible you have to publish it someone; but with GitHub forks are visible and you can see who’s forked from whom and so on.

    Personally, I don’t find this to be a very valuable or interesting thing to do at all (how do the currently existing 325 forks of Rails add any meaningful value to the project? what’s the importance of knowing who are the 2,082 current “watchers” of the official Rails repo?), but evidently many, many people think it’s the best thing since sliced bread, hence GitHub’s uptake.

  7. Sho Says:

    I think you hit the nail on the head there – the main aspect of GitHub’s popularity could well be that your fork of a project is visible, and your own little projects are visible. Although it’s easy enough for you and me to run our own Git servers, as we both do, you do need a server for that and I suppose people prefer to host their projects in a well-known central place. Basically the same principle as people writing their blogs on Facebook et al instead of individual blogs such as this and your own.

    So it’s a barrier to entry thing, and also a safety in numbers thing. GitHub’s pretty nice source browser is also a factor, I’d wager, although it’s painfully slow most of the time. I also have to give credit to their rubygems build system, which works very nicely.

    I think the “ideal” setup right now, as far as I can tell, is to run your own Git server as the “authoritative” repository and GitHub (or similar) as a bidirectional peer. Best of both worlds really. However, I simply can’t be bothered doing that, nor do I feel any need to boost my open source reputation by trying to get my projects noticed on GH or anywhere else, so I guess I’m not their target market ; )

    I confess I’d like to see Git proper’s web browser improved and its advanced functionality (commit rights, the auto-sync you mentioned, etc) made easier to use – I expect that would also spur a more distributed “ecosystem”.

    Anyway, I’m in the same boat as you – I don’t use GitHub for my own projects, nor will I. I didn’t use RubyForge or SourceForge or any predecessors either. But then again, I also run my own email, web, blog, DNS, etc infrastructure – I run my own servers as a rule, and I like it that way, but I’m obviously in the minority.

    So yeah, useful service for some (or most), but I wonder how they plan to make money from it. I would expect the free public accounts to outnumber paid accounts by a huge factor. And the obviously crippling load from the public accounts leaves them struggling with performance – GH is slow, very slow, and getting slower. Who would sign up for a paid service under those conditions?

    In fact, who would sign up for a paid Git hosting account at all? That kind of thing just doesn’t occur to me at all, although I guess someone must : )

  8. Wincent Colaiuta Says:

    I think you hit the nail on the head there – the main aspect of GitHub’s popularity could well be that your fork of a project is visible, and your own little projects are visible. Although it’s easy enough for you and me to run our own Git servers, as we both do, you do need a server for that and I suppose people prefer to host their projects in a well-known central place. Basically the same principle as people writing their blogs on Facebook et al instead of individual blogs such as this and your own.

    Funny that. I feel exactly the opposite way. In my experience I’ve been burned too many times over the years (about 12 years now) by putting my “eggs” in other people’s Internet “baskets”. So I always prefer to “DIY” whenever possible. There are already enough ways to aggregate your content with other people’s (from things like feed readers, trackbacks, bookmark sites, and Google itself) that there simply isn’t any need to be physically aggregrated and living on the same machine.

    So it’s a barrier to entry thing, and also a safety in numbers thing. GitHub’s pretty nice source browser is also a factor, I’d wager, although it’s painfully slow most of the time.

    But how much of that is actually superior, and how much of it is just a nice choice of fonts and colors in the CSS? If you compare it to gitweb that ships with Git itself, I find the latter to be more usable and more attractive (I like the “spartan”, non-”Web 2.0″ look of it).

    I also have to give credit to their rubygems build system, which works very nicely.

    No idea about that as I haven’t used it. RubyForge was pretty much the “the” source for RubyGems so that’s what I’ve always used. Never really saw any reason to reinvent the wheel.

    I confess I’d like to see Git proper’s web browser improved and its advanced functionality (commit rights, the auto-sync you mentioned, etc) made easier to use – I expect that would also spur a more distributed “ecosystem”.

    I’m pretty much sure that you won’t see stuff like that making it into core Git. Things like access control are really outside the scope of the tool as far as most Git developers think. You can set up all sorts of access control without modifying Git itself; Git’s job is to track content and not shape or constrain how you access it. Kind of like how a text editor only serves to edit text, and you decide access to files through the existing mechanisms of UNIX file permissions. You may or may not agree with it; but that’s the way the key players on the Git mailing list think and so that’s the way it will stay.

    Anyway, I’m in the same boat as you – I don’t use GitHub for my own projects, nor will I. I didn’t use RubyForge or SourceForge or any predecessors either.

    I use RubyForge to host my gems because I want users to be able to type “gem install foo” with no additional config and have it just work.

    So yeah, useful service for some (or most), but I wonder how they plan to make money from it. I would expect the free public accounts to outnumber paid accounts by a huge factor. And the obviously crippling load from the public accounts leaves them struggling with performance – GH is slow, very slow, and getting slower. Who would sign up for a paid service under those conditions?

    Hard to imagine. The thing is, if you have money to spend at GitHub you may as well spend that money getting a server (dedicated, virtual, whatever) at any other hosting provider. Obviously, there are thousands upon thousands of hosting providers, so it is a very competitive environment in which you can get very high quality service for a reasonable price. And if you’re that kind of customer (closed source), the all the “social” side of GitHub, the visible forks and what not, is completely irrelevant to you. As such, I can’t understand their business model either; it may be that their customers are of the “sucker born every minute” type.

  9. Wincent Colaiuta Says:

    On the subject of why people like GitHub, just went over to see what’s been happening there lately. I see from their blog that they’ve been making quite a few changes lately:

    For example, the language breakdown (no surprises):

    And more details for each language:

    And some nice visualizations:

    These actually look pretty nifty, although they seem a bit glitchy on the projects that I tried them on (Merb, Rails).

  10. Sho Says:

    there simply isn’t any need to be physically aggregrated and living on the same machine

    Exactly, and with super-productive frameworks like our favourite whipping boy Rails there’s even less excuse.

    I’ve actually been conducting experiments running my own IM servers, I think that will be the next domino to fall : )

    But how much of that is actually superior, and how much of it is just a nice choice of fonts and colors in the CSS?

    Good point. Maybe I could try overhauling GW’s CSS a bit. I agree GW is basically the same thing, just looks a little less “modern”. But I share your general dislike for all things “Web 2.0″.

    RubyForge was pretty much the “the” source for RubyGems so that’s what I’ve always used

    GH provides largely the same functionality. It’s useful because, and only because, a lot of interesting gem projects are using it.

    And to be honest, Rubyforge has a pretty terrible interface. No source browser at all, which is pretty slack in this day and age. I’m fucking sick of having to check out a RF project just to see its code, or open the gem from its installation directory. And some of them are using CVS!!

    I’m pretty much sure that you won’t see stuff like that making it into core Git. Things like access control are really outside the scope of the tool

    Oh yeah, definitely. And I wouldn’t have it any other way – I really appreciate Git’s approach to these matters. We have file/ssh/etc permissions already, why should Git re-invent the wheel?

    What I was trying to say was that GH does make that kind of thing easier, compared to ssh’ing in, creating users, adding them to groups etc. I would predict a 3rd-party “git manager” app, preferably web-based, would spur people into further private deployment. I’d rather administer this blog via a web interface than via editing files over SSH; I’d rather have a web admin page for git where I could see everything rather than editing config files.

    Of course, I use git regardless, just saying I think that kind of thing would make running Git privately seem less scary to the average joe.

    I use RubyForge to host my gems because I want users to be able to type “gem install foo” with no additional config and have it just work.

    Given the massive uptake of GH, and proliferation of gems hosted on the service, I would expect Rubygems maintainers to add GH as a default source soon enough.

    However, they better increase their reliability before that time!

    it may be that their customers are of the “sucker born every minute” type

    Well, there’s thousands and thousands of them too, so maybe GH isn’t in trouble after all ;-)

  11. Wincent Colaiuta Says:

    The truth is that for the likes of you and me repository access control isn’t really much of a headache. I imagine you work like I do: your repositories are either fully private and you’re the only user with read or write access to them, or they’re fully public and anyone at all can read them (but only you can write). It’s one of the benefits of being a lone coder; one less hassle to think about.

    And the great thing about Git is that the “only I can write” access pattern is no longer a barrier to participation. Someone can easily clone the repo, make whatever changes they want, and then if they want them integrated back in say, “hey, can you fetch these changes from me?”, or just send them over as patches (dead easy with “git format-patch” and “git send-email”).

  12. Sho Says:

    I see from their blog that they’ve been making quite a few changes lately:

    Well that’s all very pretty and all, but honestly I don’t care about any of that crap.

    What I want from GitHub, or any similar service, is:

    1. Good, clean web access to a hosted git repository – PASS
    2. Centralised user & access control – PASS
    3. fast speed – FAIL
    4. reliability – FAIL

    Note the absence of “kewl graphs of who commited what why when” in the above list.

  13. Wincent Colaiuta Says:

    For me the “kewl graphs” are attractive because points 1 through 4 are already nicely handled by my own server.

  14. Sho Says:

    points 1 through 4 are already nicely handled by my own server

    Me too, which is maybe why I hadn’t even noticed those features until you pointed them out.

    BTW, I’ve told you this before, but just to get it all on one page – another major bone of contention I have with GitHub is that it leverages an open source project as the absolute core of its business, and yet it itself is not open source.

    That is a problem for me. While it is certainly within their legal right, I see it as taking, and not giving.

    GitHub Actual (ie, the github.com installation) has enough critical mass now that I think it can be safely open sourced without too much loss of value. How about it, PJ?

Leave a Reply