Indeed, it started out as a debugging tool. Weighing in at 2800 lines, it's going the way of irb, giving you a lot of functionality you may never use. For a more modular irb alternative, see http://github.com/cldwalker/ripl. To attach to live running processes, see https://github.com/ileitch/hijack
It's actually built for this purpose. It's not supposed to be anything like ripl is. I'd also recommend using that as a lightweight IRB alternative. But Pry isn't supposed to be that.
Also, Pry is currently weighing in at less than 2 thousand lines. It's not going the way IRB is going, but it is indeed very feature full, and probably will contain stuff you may not use every day. It's also easy to extend, so moving commands out of core and abstracting them into their own external libraries would be simple, would we ever want/need to slim the library.
Is being "large" and feature-rich a bad thing for a command-line tool? I don't want gems with unnecessary features bloating my projects, but I have no issue with feature-rich gems that help me develop more effectively, even if I don't use all of the features present.