I think later on, they will probably really appreciate if you took care of the baby while they go out on a date, every so often. Of course you need to be able to go on dates with him as well! But that would probably be very good for the both of them.
At the beginning though, I assume she'll want to be with the baby as much as possible. Even if you don't go to their place often, offering to do errands would probably be priceless to her. It might feel awkward for you, but they're even less likely to ask you to even if they really need help! So I would say there is no harm in offering.