There are minerals throughout central Pennsylvania from Harrisburg south to the WV and MD borders. Road cuts on county roads can be interesting, though you'd want some sort of guide to identify PA minerals. The problem is finding public land where you can look. Schuylkill County is famous for its quartz crystals, and of course there are old coal mines that you may be able to access (hard anthracite was used as a gem during the Victorian period, and there are plant fossils that might be found in the tailings, too). Some quartz north of Philadelphia, too (streamsides in parks in Montgomery County), though I don't know whether there are restrictions.
Sandstone outcrops and roadcuts in the southeast part of the state have yielded trilobites and other marine fossils. Swatara State Park north of Fredricksburg has a fossil pit where you are allowed to dig. The Montour Preserve east of Muncy also has a fossil dig. The state has some info
here.