Yeah, you can't do it by voice. That lady wants to know the whole address, or no go.
One neat trick I use now (but only when I have the whole address, which won't help you I'm afraid) is to cut off that witch as soon as she starts talking by pushing the Voice button on the steering wheel, bypassing her tedious conversation and getting straight to the next prompt. This has cut the time it takes me to enter an address by voice by a good 2/3, as opposed to waiting for the next prompt after she is done talking, which takes forever as we all know. Try it out next time.