Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Rails 3.2.13 - Performance regressions and major bugs (bugsnag.com)
125 points by sephine on March 20, 2013 | hide | past | favorite | 72 comments


Okay, some of these problems are (unintended) direct side effects of the security fix bugs.

Like I the one about scopes that effected Github -- github quoted [this commit](https://github.com/rails/rails/commit/f980289fd2c1b9073a94b5...) as introducing the bug, which is the commit meant to address CVE-2013-1854.

So, okay, bugs happen, even with security fixes, I can forgive bugs.

But others of those performance regressions... " handing that task to Sprockets instead of resolving internally, which seems to be the cause of the performance issues."

This doesn't sound like the unanticipated side effect of a security fix. This _sounds_ like an independent change they threw into the same patch release with four (four!) security patches.

Why is Rails putting security fixes and other non-security-related enhancements together in the same patch, so you can't get the security fixes without also getting the possibly buggy new features/refactors? Does Rails actually do this by policy? Was it a mistake?

Bugs happen, including bugs in security fixes, sure. But it's BECAUSE bugs will happen that you don't throw unrelated new features or refactors into the same patch release as a security patch, so the only way to get the security patch is to risk MORE problems from the other stuff too! Come on!


This is a weird comparison, but they should adopt the same release policies as Drupal. Security releases are done separate from maintenance releases, even if they are released at the same time.

That allows you to provide the security releases as a separate patch file so you can apply individually. You can skip maintenance releases that cause problems but still keep things secure.


Here's a pointer to the security fixes in the release as a monkeypatch to 3.2.12, without any of the other cruft:

https://github.com/discourse/discourse/blob/master/lib/freed...

The discourse folks who threw this together have an app (Rails-based forum software) that was hit very hard by the performance regressions.


The rails team already does this. Here's an example patch for one change:

https://groups.google.com/forum/#!msg/ruby-security-ann/o0Ds...

Unfortunately that wouldn't have avoided this issue, as the problem was actually with the small security patch.


AFAICT Rails does point releases by just taking whatever's at head on the 3.x master branch, building a release candidate there, and iterating until nobody is reporting bugs.

This isn't a security patch - a stable branch with deliberately cherrypicked fixes - it's just a plain old "point release." And as is to be expected with Rails at this point, point releases break apps.


And as is to be expected with Rails at this point, point releases break apps.

As a primarily Django guy with an interest in Rails... is this hyperbole/overstatement or is this a common perception?


It's not particularly common in my experience. For example I upgraded several apps to 3.2.13 this week after tests, and haven't seen any breaking changes, and most of those apps have come up from 2.x without unexpected bugs (though I think I did have to make changes for 3.0 before going into production). YMMV, and I might have missed a subtle bug caused by this change but have checked since the github post and can't find any issues.

This bug looks like it was caused by the interaction of Github's scoping with a small change to Rails to attempt to unify the query building a little and fix a potential security issue. Where queries are merged where they apply to the same column, but (probably unintentionally) they were not merged when one was a string and one a key, which happened in some circumstances with scopes. Would love to see a slightly clearer explanation of this bug - the article doesn't provide one.

They could (and usually do) isolate the security patches in a separate release with just minimal changes - at least that would make the process more manageable, but unfortunately in this case I don't think that would have helped, as the change was directly related to a change for security and just had unintended consequences.

In fact I think from their blogpost github merged the CVE patches separately after testing rather than using the update, so having a separate security release wouldn't have helped them either, it's just an unfortunate confluence of circumstances for them due to a gap in testing (perhaps on both the Rails side and the Github side). See

https://github.com/blog/1440-today-s-email-incident

The only lesson I draw from it is that perhaps Rails could slow down their release cycle a little, issue more RC builds, and encourage large users like github to try them out extensively in development environments (hard to persuade people to do this on small point releases though). Up to now they've done a pretty good job IMHO and I've not seen many issues crop up, particularly with security releases, which are usually straightforward.


It was certainly the case on the 2.3 "stable" branch two or three years ago: http://david.rothlis.net/rails/experience_report/#stable_bra...


From my own anecdotal experience it is true. The ruby gem responsible for most point release breakages affecting me is probably activerecord.


I'm also a Python dev, and my observation from several threads here (and elsewhere) is that the perception is quite common.

IANARailsDev, obviously.


At this point I'm seriously considering dropping Rails as my framework of choice and reverting back to ASP.Net MVC.

Why?

1. Security fixes are released very quickly (good thing), but more often then not they break existing code (bad thing) - and while you _can_ wait for the next patch to fix those break points, you're left wide open since everybody can see what was broken and how to exploit it.

2. I'm not nearly smart enough with Ruby to apply 'band-aids' I read people write in Rails source code. I just don't have the time and energy to fix code in the actual framework when I need to be fixing _my_ code.

======

So there you have it; I'm bailing.

It was a fantastic ride, Ruby is a beautiful language, but Rails is just a clusterfuck (_for me_).

I need stability and predictable behavior. If that means releases every 6 month to 1 year, so be it.

So long and thanks for all the gems.


I've been running Java and Rails since 2006 in my company (large), we use C# for a small number of desktop applications.

This will come out harsh, but I'm being honest: If you are not willing to fix/adapt/improve the framework you are using, you better not be using ASP.NET also (try a google search on asp.net issues), or any framework at all. In fact, developing larger more complex software might not even be possible without adapting frameworks for your specific problems.

That said, our run so far with Rails and Java (we are still using 2.3!!!):

- We have 38 patches applied to Spring

- 4 patches in apache santuario.

- One security patch on Grizzly (glassfish)

- Rails: One single pull request https://github.com/rails/arel/pull/174


It's really nice being able to trace into the guts of hibernate, spring, etc to see what the heck is actually going on and fix it if it's broken. When I used to do Microsoft stuff, I'd run into bugs in the stack that I couldn't fix. This was enormously frustrating and time wasting. The typical advice given out by support was re-install or buy the latest upgrade. There were also a lot of deliberately imposed architectural limitations to prevent you from working around lack of "enterprise" features in the basic versions which were incredibly frustrating and the enterprise version pricing was astronomical.


The open-source situation at Microsoft has really improved. Source is available for a lot of their web stack, contributions are accepted, and there are some nice open-source frameworks (like Nancy, MIT-licensed, inspired by Sinatra) that run on IIS or another Owin-compatible server these days.

I generally prefer Ruby-based things myself - this post isn't trying to sway anybody to the MS side; just wanted to give a bit of credit where it's due and perhaps a bit of news to those who haven't looked over onto that side of the fence in a long time.


>If you are not willing to fix/adapt/improve the framework you are using, you better not be using ASP.NET also (try a google search on asp.net issues), of any framework at all. In fact, developing larger more complex software might not even be possible without adapting frameworks for your specific problems.

Huh what? Stack Overflow with it's extreme load works quite well with ASP.NET.

>try a google search on asp.net issues

Okay, I did, what I am supposed to be looking at?


The point is, SO almost certainly aren't using vanilla ASP.net. Nobody can succeed for too long without maintaining and cultivating their stack.


We're on ASP.NET MVC4, a stock build.

We've configured route registration a little bit specially* to speed up some common cases and we don't necessarily use every feature in the framework; but we're far from a proper fork.

ASP.NET MVC performs well enough that we get more bang from focusing on our own code basically.

*This amounts to registering our highest traffic routes _first_, and checking left-hand matches before Regex'ing (since we have a ton of "/foo/\d+/bar"-ish routes).


I don't know about Vanilla ASP.NET but they are pretty much using stock config'd IIS and Servers. So I would be very surprised if they weren't using stock ASP.NET (with their own filters, etc using public APIs I'm sure)


Forgive me, I'm not an MS person, but is there no opportunity to augment/customise/replace/mix-and-match/etc the ASP.net components?

From what I've read on http://highscalability.com and the like, they love to tweak.


Wow, you're either one of those incredibly rare ruby devs who actually run windows, or you're going to switch your entire environment in order to switch frameworks. Something doesn't smell right


I'm a polyglot. I don't encase myself in any 1-thing. I use Mac at work, Windows at home for media and games, and Linux Mint 13 on my laptop for Ruby/Rails. I only used Linux because of Rails, but since I'm switching back (I have Visual Studio 2012 downloading as I type this), I'll have no need for Linux on my end-user machines anymore.


If you do not know your language/framework well enough to adapt it to your circumstances, you certainly are not well qualified enough with said software to list it as something you "know".

This is one of the largest lies perpetuated on this site. If you know just enough of something to follow some tutorials and crank out cookie-cutter sites using Rails/Django/Asp/Grails/CodeIgniter/etc, but fail the moment you encounter basic ecosystem problems, you aren't a polygot. You're a liability to your team for not understanding anything well enough to work around common issues. You're probably curious and love to try new things, but the implied part of being able to proficiently wield whatever tool is correct, or at least on hand, is something escaping you.

If by polygot you mean you know some syntax, the basics of the ecosystem, and generic system design, algorithms, etc while expecting to dive as deep as necessary into any given environment, you're still failing the definition. This would be the point where you RTFS so you know where to override/monkeypatch/workaround.


I don't know who you are, but this comment embodies the bad side of HN. "You" is in the above comment 15+ times - it's so personally insulting and unnecessary. Belittling some stranger's worth to their team is unproductive, toxic behavior. I'm upset I can't downvote this into completely matching the site's bg color.


+1 to you and -1 to the aggressive and insulting comment that prompted you to comment.

I see this a lot on HN too: people who've experience some level of success (as pg would say the first thing you learn when you get rich is that there are many levels of rich : ) and who hence think they know it all about everything and can constantly try to diminish others.

There's a lot of negativity here but, thankfully, there are also others who are here to share, educate and learn.


I disagree. I count writing and selling commercial software with said technology as 'knowing' something. Academics be damned.


Would you really be willing to give up the productivity benefits and developer happiness over a few mishaps? Also, it's generally a good idea not to upgrade to the next software version until seeing the repercussions its had.


That's the thing! I find myself ping-ponging in my mind between "But Ruby and ActiveRecord are so beautiful and easy to use." and "But what happens if the 'magic' breaks?"

Trust me, I have been thinking about what you say every day for the past 5 days.


You're thinking too much. Unless it's a "the world is ending" level security fix like the recent yaml stuff, wait a couple weeks for this stuff to come out and get fixed. Then upgrade.


You do know there are other Ruby frameworks than Rails, and other development environments than ASP and Ruby right?


Yes, I used to use ASP.Net MVC for pet projects, then switched to Rails - I also use CakePHP and Django for actual paying work/contracting.

But thanks for the passive aggresive comment.


While that may be a passive aggressive comment, sinatra is bloody brilliant all the same. You can basically pretend you're writing rails code, but with a tiny fraction of the dependencies, and actual fast response times.

Sinatra, sequel and postgresql is a dream combo. I wouldn't use rails for a new project these days, I got bored of the un-navigable code a while ago. These patches and regressions are the icing on the cake.


> Yes

Because your comment didn't exactly hint at it considering its completely binary assertion of "Rails or ASP"

> But thanks for the passive aggresive comment.

I'm very sorry you found it passive, that was not intended.


You're on the cutting edge of an extremely popular framework.

If you want, you're welcome to use Rails 2.3 or older and you'll encounter many fewer changes. I imagine the 3.0 branch will have much the same stability once major development moves to 4.0.


FWIW, the Rails security team says they're about to drop suport for 2.3, and this time they really mean it. (They say.) The current announced security policy is that once 4.0 gets released, they'll only be doing security patches for 4.0 and 3.2; 2.3 will be left out in the cold.

It's quite possible that someone will pick up the baton, as there still are a number of production 2.3 apps out there, and porting to 3.0+ --- more a "port" than an "upgrade" --- is a real pain. But I'm not sure anyone has stepped forward yet, and until someone does, you're taking your chances.


You should seriously consider running mono>=2.10 with asp.net mvc. It completely fixes your rails security issues, while still letting you use ruby as your main languages.

Also you gain a _lot_ in security by obscurity and hipster points. I don't know about stability, haven't tried this myself.

Check out the mono project for details: http://www.mono-project.com/Mono_Project_Roadmap


I've not heard of anyone using IronRuby in earnest, and it doesn't seem to have been updated in a couple of years. Is it a realistic option?


You are now in the place I was recently -- the decision we tool was to migrate onto Python and their web ecosystem.

We haven't looked back since -- well, we have given an glance or two back since, and viewing the slow-motion car crash that is Rails today, we just feel sorry for those left behind that are discovering 'The Rails Way'.

Ironically, given DHH's comment on Rails about it being 'Omakase', I should point out that it translates as "I'll leave it to you" -- security, I'll leave it to you, software engineering, I'll leave it to you.

Food for thought indeed.

Remember -- Ruby to pose, Python for Pros.


I disagree with the final statement of this, that a "security only" release would have somehow helped the situation. I feel like part of the problem is that things are being merged into 3-2-stable and nobody is using them because they're never put into a release.

3.2.11 and 3.2.12 were both security-only releases, and so fixes and patches that were not security-related have just been piling up for the longest time.

Seems like a "release often" mindset might be better and might ensure that things are actually being used. Sure, I know someone is going to blast me saying that bugs like this shouldn't make it into 3-2-stable at all, and since this is Hacker News so whoever says that knows they're ten times the developer that everyone on the Rails core team is. But the reality is that bugs make it into stable branches, and almost nobody is running against the stable branch (Rails core people are probably mostly running against the 4.0 branch, everyone else is running against the latest release).


Actually 3.2.12 also added some sort of silly (and broken) patch related to quoting numeric values in ActiveRecord. The primary thing I want out of 3.2.13 is reverting this.

To your more general point, while I don't disagree with your diagnosis of things piling up in stable, I'm also not a big fan of your proposed solution. It looks to me like 3-2-stable is getting filled with a lot of stuff that just should not there; increasing the release frequency is not going to solve this problem.


I think the only mistake made with this patch release was merging in 250+ commits, ranging from last year to the present day. Maybe that's standard for such a large project, but it feels like a hell of a lot of tweaks for a minor release.

As for the rest - I think the issues in January has put Rails in the spotlight, and while scrutiny is certainly warranted, I think the extra pressure on the project over the past months to clear this stuff up has had its consequences.


I think the problem is that they do too many patch releases and not enough minor releases. This could easily have been a minor.


296 commits for a security patch is silly; What the shit tenderlove.


Seriously. Usually Rails feels like riding in an Uber, but this release felt like Crazy Taxi.


Not happy? Do something about it: try release candidates, report bugs, and help triage and fix reported bugs.


I completely agree with this. We (bugsnag) are running on rails 4 beta right now to help iron out any production issues before that is released.

We didn't, however, realize that "patch" releases also have release candidates. I'd recommend following the rails blog (http://weblog.rubyonrails.org/) which announces release candidate builds too.


Thanks for the blog post - and I didn't realize that patch releases also had RC.

It looks like this release was quite fat, for a patch release.

I added "check rails diff" to my deploy todo list.

EDIT: I just noticed that http://weblog.rubyonrails.org/ doesn't have a way to subscribe by email. I signed-up by email with this site instead http://blogtrottr.com, in case it helps other here.


You can also subscribe to the rails-core mailing list, all betas, RC and full releases get announced there.


There's quite a bit more discussions on rails-core apart from RC and releases, so subscribing to the blog is a good idea if your time is limited :-)


Ok, yes, that's an extremely great and open-source way to handle this.

But seriously, can we at least expect moderately stable releases? I do not think that's too much to ask, especially for a framework which has, in the past, achieved that.


As someone who is relatively new to Rails can someone explain to me why it seems to update more than anything else I've used?


Mostly to give people on forums something to bitch about.

Serious answer: a few months ago, some serious vulnerabilities were found in Rails. This apparently had a "blood in the water" effect on security people, who started dissecting the entire thing. Predictably, more vulnerabilities have emerged and rapidly been fixed.

All in all, it means that things are improving because it's getting a ton of attention.


This. I think the rapid release cycle of Rails is a good thing, rather than a bad thing. The past few months, some serious vulnerabilities were found in Rails, and I guess this made the Rails core team a little anxious.

One might argue that these performance regressions are unacceptable, and they are. I agree. Still, Rails is relatively young framework and these things happen. I expect any framework that is widely used will see some serious vulnerabilities in the next few months/years.

By the way, a good explanation of what these past Rails security issues mean is explained on patio11's blog (http://www.kalzumeus.com/2013/01/31/what-the-rails-security-...).

OT: I'm also writing a book on upgrading your Rails 3 app to Rails 4. With these rapid releases, a lot of things change and it's not always clear what has changed or what is new. Feel free to check it out at http://upgradetorails4.com/.


Bad timing. The JVM has had an abnormal increase in security patches over the same time frame too.


You don't have to stay on the bleeding edge. We're stuck with Ruby 1.8.7, and we've got a mix of Rails 1, 2, and 3 apps. For better or for worse.


You know when you're already having performance issues on Heroku having a performance regression is not fun.


How is this

> […] handing that task to Sprockets instead of resolving internally […]

even remotely related to performance issues because of random routing? You're mixing two issues that have nothing in common.


1) Grandparent has performance problems on Heroku.

2) The Rails release has performance regressions (i.e. in at least some scenarios is slower).

In situation (1) you don't want to add further slowdown so (2) is a bad thing.

Grandparent doesn't even blame his Heroku performance problems on the routing issue (although it is a plausible guess). Actually effect of the routing problem can be significantly worsened if even a subset of requests are slowed down.


I'm under the impression that the Sprockets regression should not affect you in production, because your assets are precompiled.

Is that not the case?


To be honest I don't know and I wouldn't have negatively responded to a comment indicating that or questioning it. However reading the article it sounds like it is an issue resolving asset paths so may affect you even when assets are precompiled.

Performance Regressions

There are performance regressions in 3.2.13 for both view loading and asset loading. Rails 3.2.13 changed the way assets paths are resolved, handing that task to Sprockets instead of resolving internally, which seems to be the cause of the performance issues.


Actually, the performance degradation to the precompiled assets is minimal - for a typical production environment it only effects your asset compiling time as the assets are served statically from your public directory.


The asset path and sprocket performance hit doesn't seem to affect most Rails apps, they precompile their assets, so that shouldn't be a problem.


But it does slow down dev, which doesn't not suck.


There's been a few people suggesting that, if you are bitten by a bug, you fix it and submit a pull request. That's a good idea. I did exactly this to a regression introduced in 3.2.9, complete with a unit test and a pull request against 3-2-stable and one against master. It took two months to get anyone to look at it, and while one of the two pull requests has a thumbs-up, it's still not been merged. Admittedly, this isn't a security issue as far as I can see, but still.


I personally read every single issue and comment filed against rails/rails. I often don't comment if it's not a part of the codebase I'm not familiar with, and it's the same with merging.

Can you point me to the specific ones so I can look at them again?


Yeah, tvongaza is correct. Admittedly, it's not a _big_ issue, and perhaps I'm going about things the wrong way. If you could tell me a better way of getting this issue fixed, I'd love to hear it. The only other time I tried, it took more than two years. Luckily, it was pretty easy to monkey-patch our own code to deal with this, once it broke our live site.


Other than tenderlove and (sort of) myself, nobody is paid to work on Rails. This means that it's an entirely volunteer effort, and people's lives are busy, so sometimes, tickets take a while to merge. I try to stay on top of it actively and thing still sometimes slip through the cracks for me. It's not that you're going about it the wrong way, it's that projects with ~200 open pulls are slow to merge, sometimes. Sorry it's taking so long with this one. :/

I'll bump this up in my personal list; I should really learn AR better anyway.


Pretty sure Chris is talking about: https://github.com/rails/rails/issues/8856


Ah yes, ActiveRecord is one of those parts I am _not_ an expert in.


Since when do Rails developers worry about performance?


I'm sure that Rails is great and powers many websites and that many devs loves it.

I'm also sure that people are working hard to fix these recent security issues and perfs issues.

But as a non-Rails dev I can tell you one thing: with all the attention that Rails got lately I'm sure I'll never be learning Ruby / Rails.

I'm into Clojure right now. Next target is Go.

Sadly all these Rails exploits do have a negative effect for Rails: I'm not saying this to criticize Rails. I'm saying this because I'm honestly 100% sure I'll never ever be doing anything serious with Rails and I know I'm not the only one in this case.

So, Rails devs, fix this and fix this rather sooner than later because every exploit is missed opportunities.


I don't write Ruby either right now, but I would suggest you not discount it as a nice language because of your perceptions of Rails.


When you bring up Clojure, I think it's fair to point out that, beyond even the default-unsafe reader thing brought up a couple weeks ago, the JVM itself has had several fatal security issues recently. No platform is completely without bugs.

That said, I don't really know if RoR has had more than normal or whether they're just more widely talked about. It does have a huge and vocal user-base.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: