Is knowing what notes are on what frets important?

I’ve been learning guitar for a while now. and currently my teacher is teaching me blues improvisation, i can play pentatonic scales in every mode, but I still don’t know what note is on what fret. I’ve brought that up a couple of times to my teacher, but it seems that it’s not really that important, should I try and learn where each note is or will I just naturally learn with time? I think if I knew where each note is, it would help me improvise better and not rely on just scale patterns.