I'm not a fan of great complexity in RPG puzzles. If you've ever played Legend of Zelda: OOT and tried to muddle through the Water Temple, that's a great example of a very complex, multi-layered puzzle that is not fun if it doesn't click with the player. That especially sucks when you have to solve the puzzle to proceed.
There's a philosophy in RPG encounter design that GMs should invent a puzzle without a particular solution in mind. Much of the OSR movement is about creating scenarios for the players to interact with, and then remain reactive instead of prescriptive. This is the same idea and it encourages the GM to rely on players for solutions, which in my mind is how RPGs should operate. Let players go where they want; the GM is just there to hold the flashlight.
But that doesn't mean you can just throw your players into an empty room and let them flail around. Just like any other quest or encounter, you need to give them hooks. The difference is that your hooks may not be attached to anything. Maybe you decide that later when they pull on the line.
1. Give players a tutorial
Video games have improved a lot over the years in terms of tutorials, incorporating tutorials into gameplay in unobstructive ways. People expecting to have fun playing a game don't want to read through rules or have their experience interrupted with a tooltip.
When video games introduce a new mechanic or enemy, they will often show the player the new thing in isolation. A new kind of platform crumbles when you jump on it. Good design lets you encounter it without risk, see what it does, and then it gets incorporated into the gameplay loop. No need to stop and explain. Let the player investigate and figure it out for themselves; that is way more rewarding.
In a recent adventure I ran for my (yet untitled) d6 RPG (heavily based on Troika!), the players were trapped inside a dungeon by the psionic power contained in crystal orbs. In the second room they entered, I described the presence of such an orb. The doorways were blocked by this strange psionic force. When they tried to bash their way out, the orb flashed and flared with each blow against the barrier. When they smashed the orb, the barrier dropped and they could proceed.
I made it very clear what things were in the room to interact with and what was stopping them. Their actions allowed them to get clues (the orb flashed when they hit the barrier) to show the two things were connected. The fact that this force was psionic gave them insight into the nature of the other challenges they would face.
By completing the
tutorial, they learned that the next
time they encounter a barrier, they need to smash an orb.This may seem
like a no-brainer, but a lot of gameplay time is wrapped up in players
investigating a new puzzle or obstacle before they take action. If they
know what to do, they are more confident (and faster) to take action. What's more satisfying than solving a puzzle? Feeling smart and vindicated by applying knowledge you learned from hands-on experience.
2. Keep puzzles simple
In the example above, I traded player agency for simplicity. Trapping players is rarely good practice, but this forced them to interact with same puzzle they had to solve in order to escape the dungeon (their end goal).
A more complex scenario might have made this an optional challenge, or allowed them to bypass the barriers or ignore them. However, that risks that they never learn how to complete the puzzle, or they never perceive it as important. In a 4 hour one-shot, that's not a risk I'm willing to take.
In my years of putting dungeons in front of players, I've learned that less is more. Planning clever riddles and word puzzles can have diminishing results; some puzzles that have taken me hours to arrange and prepare were solved in just five minutes. Others that I thought were throwaways have taken players hours to get past.
But that's the way it goes. Sometimes players have big expectations and discard solutions they see as "too stupid" or "too obvious." Sometimes the player with the solution is reluctant to offer it, or isn't able to because the other players don't give them an opening to speak. Or maybe they roll like dogshit and you learn the hard way why randomness should not part of a puzzle.
I think that a perfect puzzle has stages and allows the players to
advance, feel great about moving ahead, and then makes them feel
reinvigorated for the next stage. This is a fantastic way to build
suspense and then release it, and once in a blue moon a puzzle will
achieve that organically. But it can also backfire and lead to a stagnant, no-fun slog. You need to know your players and their level of patience; you need to know what kind of puzzle excites them. Sometimes you just need to get lucky.
In my experience, you are overwhelmingly better off with something simple and open-ended so players can find many solutions. Bonus points if it fits into your universe and allows the
players to learn more about the world as they interact.
3. Step in when necessary
After giving the players a
tutorial (Smash orb, barrier drops) I thwarted myself in the next room.
They found the dungeon exit and
saw it was blocked by another barrier. Their first instinct
(rightly so) was to find an orb and smash it. Problem was, the orbs they
had to
smash were in different rooms. This urge to repeat the problem solving
that worked last time
caused them to lose sight of the other options available. I made efforts to drop clues that this barrier
was "much larger" than the others, but I could have been more obvious at
the risk of breaking immersion, which in a one-shot is acceptable. However, this gave the players time to roleplay among themselves, which is always good!
After two "rounds" of players investigating the room and interacting with the NPC there, I noticed that the players were losing sight of the task at hand. I think a player should never be afraid to ask "what are we supposed to be doing?" That tells me as the GM that they are out of ideas, it's been too long since the plot moved forward, or the goal is not clear. My mistake was relying on the obstinate NPC (that the players mistrusted) to tell them to "take a hike," which of course they wouldn't do; this was too subtle a clue. I stepped in and as the GM suggested that the players find a different room to explore. In a one-shot this is pretty forgivable behavior; I want them to move forward and have fun.
It never fails that a puzzle the GM thinks is stupidly easy can hold up a table of full-grown adults for hours. Be aware of your pacing and try to cultivate a feeling of forward momentum. The fun stops shortly after progress stops.
Like a concierge at a theater, turn on the flashlight for a moment and show them the right direction. They're not going to focus on that little ripple of immersion breaking and they're not going to remember it at the end of the session. They will have their little "why didn't we think of that?" moment, but their focus will be on the action, on getting where they want to go.
Of course, when possible, the GM should deliver this hint some other
way, like through an NPC or flashback. Remember
the letter the king sent you about getting his daughter back from the dungeon? That's why you're in the dungeon.
4. Encourage curiosity
I will ring this bell again: the GM's job is to put a challenge in front of the players and let them explore. There was a fantastic metaphor used a few months ago on the RPGBot podcast: the GM provides a description that's like a tapestry with lots of little threads. The players will pull on those threads to get deeper into the story. Like a pojnt-and-click adventure, a good description sets the scene and makes it clear which features, objects, and NPCs are there. The players then interact however they can imagine, and pull the string until they move the plot ahead or their curiosity is sated.
With puzzles and mysteries, sometimes putting out the right clues--remembering to tell the players that there's a thread to pull--is essential. In my notes, I always need to give myself reminders so I won't forget those important details, like a bloodstain that will lead the players in the right direction, or an ominous hum that foreshadows some magical ritual. This is awesome when it's atmospheric and woven into a good description. If you have to tack it on after ("Oh wait! There's also a baleful, unearthly glow coming from the south tunnel.") that's better than nothing, but at least you didn't forget it.
Likewise, in the barrier-orb puzzle, the players had other options beside destroying the orb. They had a caster in the party who could have potentially taken control of the orb to drop the barriers. I would have let them proceed by trying to push the orb through the barrier, or even allowed the Skeleton Jelly character to slip past with a good Contort roll. From a game design standpoint, it would be interesting to switch something as fragile as a crystal orb with another object (like a pipe organ or a mushroom) to see how it would change expectations and problem solving attempts.
An effective GM needs to be prepared for the crazy ideas the players suggest. It's much easier to be open to this when there is no "right answer" the GM is expecting. If the door only opens by pulling a lever, I guarantee you the players will try everything but the lever!
Creative solutions are the heart and soul of roleplaying games. The stories we bring with us after the session are always about the unexpected things. The more you encourage the unexpected and remain flexible with your players, the more fun stories you will create at the table.