Luckily, Jan Dudek (thanks!) came up with some nice solutions. The best of them uses ActionDispatch::Callbacks in combination with require_dependency. Here is an example:
ActionDispatch::Callbacks.before do require_dependency "app/models/events.rb" end
I have tried it in one project and it works pretty well in all environments.
Why would I want to keep many classes in one file?
I agree that this technique breaks the Rails convention of keeping one class per file. Possibly, it can confuse some developers on where to find classes definitions. In this case it's important to communicate with the team whether it's worth trying and explaining how it works before using it.
For me this technique is useful when I have many Rails models, all very short, related to the same feature. The example above (app/models/events.rb) keeps definitions of Event, Attendee, Attendance, AttendanceType, EventBoard. All of them are short and instead of jumping between many files I've got one file which keeps it all in one place. It's hard to measure whether it's good or bad, works well for me so far.