'54 Hornet turning signal housing removal

I am trying to remove the housing for the turning signal assembly from my '54 Hornet and wanted to see if anyone had a great idea for this. It looks like it was pressed into place and needs to be bumped up to remove, but I don't want any damage. A rubber mallet may be the thing to use, but ???? See photo.