That's always something that's really important for an actor - to find an opportunity to do a scene where there is a moment like that, where you manage to connect with everyone.
I mean, you still can't jump offstage and go read a book. But I'm getting better at it. It is something you can manage. You can still give everything you have to the audience onstage, and have something for yourself.