we will all be in some variation of jeans/shorts, t shirts and sneakers

The restaurants at the Inner Harbor are used to casually-dressed tourists and hordes of people in outrageous costumes because of Otakon, so I don’t think you really need to worry about your group’s attire. Just avoid fine dining restaurants like the Rusty Scupper and a couple of $$$$-places in the Harbor East area (such as Charleston), and you’ll be fine.

You might want to try Bubba Gump Shrimp Co. if everyone enjoys/is not allergic to seafood.

I thought the food at Bubba Gump Shrimp Co. was mediocre, but that’s just me.

I forgot to mention Fort McHenry National Monument as another tourist attraction to visit in Baltimore. I felt my visit to Fort McHenry was worthwhile, but I enjoyed the American Visionary Arts Museum more, in part because I like art better than history (and partly because I'd rather have "America the Beautiful" as the national anthem than "The Star Spangled Banner").