Gems can even be received from your Lady in White at Tatsuhime Shrine at nighttime when you have offered her enough of your fish she requests. Alternatively, the social gathering can bypass the puzzle completely by teleporting a party member previous the stone door through Misty Step, Dimension D