For a lot of people, D&D is the first tabletop role-playing game that they will be exposed to. This makes sense, because it is the most ubiquitous TTRPG system, with it being sold at certain major retailers (thanks in part to the distribution network that comes built-in by being owned by one of the biggest toy companies on the planet). Add to that all the streaming groups who play with that system and you have a massive amount of market coverage.
But it isn’t the only game on the market. There are hundreds of other TTRPGs and story-games available, which can be a bit overwhelming, but there are also a lot of people out there who will gladly give their time and energy to help you narrow down your set of options to what you are looking for in a game.
Because (and this may be controversial to some), D&D is not actually designed to do EVERYTHING. It is, at its core, a combat-game, and always has been. But that may not be what you want from your game.
Maybe you and your table want to have a game set in an existing intellectual property (IP), and you do not want to attempt to “hack” D&D until it will work for that setting. Good news! A lot of IPs already have a dedicated TTRPG. The thing is, some of those systems are very limiting, because they are designed around that IP in tone and possible actions.
Then there are other systems that are built around an original IP, or systems that are setting agnostic with an original IP constructed around them.
Say you want to run a game in an existing IP, but there is not dedicated system already produced? Which system do you use? My advice is you look for a system that matches the tone of the IP. If the IP is grim-dark, you need to find a system that is focused on grim-dark. Do characters have plot-armor in the IP, or is the IP lethal to main characters? There are some systems where death of a player character (PC) only occurs by tacit agreement between the player and the game master (GM), but others where PC death can occur because of one bad die roll. This all matters significantly, because taking a system designed for high PC death and “nerfing” it to provide PC plot-armor starts to unravel the way that system operates mechanically. The same is true with taking a system where PCs have built in plot-armor and introducing lethal rules. There are some systems where combat rules are tertiary to the system, and not well defined, because combat is supposed to be rare or non-existent as a conflict resolution in that game system.
Example 1: the old West End Games Paranoia “system”. I put “system” in quotes, because this game used what I would refer to as an anti-system, because nothing matters. The “system” is a facade to facilitate the setting. (Full disclosure, I have not played the current iteration of the game by Mongoose Publishing, so this example is limited to the old WEG versions.) This anti-system really only works for the specific IP, because the rule mechanics are intentionally designed to be nonsense. This game encourages the GM to do a lot of head-nodding and to look at table “XX” as part of the illusion that there are rules, before telling the player that their action failed or failed miserably. “How is that fun?” you may ask. It is a legitimate question. The answer is that it is more fun if everyone is actually in on the joke to some degree or another. The rules are not what makes the game enjoyable. It is also why the rules do not really work with anything outside of the related IP.
Example 2: the old West End Games Star Wars system (1987). This system was built around the first Star Wars trilogy, and is designed to facilitate adventures in that specific IP setting. Overall, I think it did a decent job, while having some massive gaps in the rules (Force powers were extremely minimal, because no one was supposed to be playing a fully-fledged Jedi, because the Jedi were all but extinct during the time period the game was set in). The D6 system it used would also be used for other WEG games like Ghostbusters (1986, with a 2nd edition in 1989) and Men in Black (1997). So, the underlying system of Star Wars (D6) would be considered system agnostic, and it did not focus exclusively on combat or non-combat. In fact, WEG would eventually just release the D6 System on its own as The D6 System: The Customizable Roleplaying Game (1996).
Example 3: Dungeons & Dragons (any edition). A fantasy setting combat simulator with non-combat peripheral rules attached. It works fairly well if you are fine with its very delineated character role functions and strictly defined magic system. There are some often very vague “optional” rules (depending on edition) that allow the game to go in a less combat-oriented direction, but the character roles are defined by their various combat abilities. Even the majority of the spells are centered around how they affect combat. The 3rd Edition set of rules was the closest that you get to being setting agnostic with the D20 system, especially when you look at the variants of D20 Modern and D20 Future, although some would argue that while using a similar rules framework as 3rd Edition, they were completely separate systems. So, D&D may work well for your swords and sorcery fantasy setting (maybe some tweaks required), but may not work if your game is based upon comic book superheroes (or only work with a significant amount of “hacking” that effectively creates a fundamentally different system framework in the end).
“That’s a lot, and you haven’t actually given me an answer to the initial question? What system should I use?!?!”
Sorry about that. I’m not skilled at prognostication. Start out be looking for an online community of TTRPG players. I recommend Twitter, and just putting the question out there tagged with #TTRPG and “looking for recommendation/advice”. There are a lot of gamers who LOVE answering these questions, but look for the people who are asking follow-up questions on what type of game/genre/tone/mood/etc. you are going for. They are asking these questions for a reason, because it helps them narrow down their recommendations. Trust me on this one, because there are hundreds (no exaggeration there) of games available, and some of the folks on Twitter are very familiar with a LOT of them.
(For instance, I am not terribly familiar with a lot of the Indie story or narrative games. They aren’t really my cup of tea. But there are people with an intense passion for these types of games on Twitter who can give you some very good advice.)
You can also try out your local gaming store, if you have one. Experiences will vary. Some of these stores are SUPER friendly with amazing staff who are very knowledgeable about the subject matter. Some of these stores are owned/operated by elitist gatekeepers who will respond to your questions like you are ridiculous for even asking. Don’t shop at those stores.
Fair warning, TTRPG space on Twitter has its fair share of gatekeepers and grognards (veteran players who believe that their 10/20/30/40+ years of gaming experience make them the FINAL opinion on any matter gaming related, and if you disagree with their hallowed opinion, then you are some sort of upstart that doesn’t honor their elders). General way to spot these sources is that they will TELL you what to play, as opposed to inquiring what you are interested in and then making informed suggestions to you. You will also get people making joking references to Rifts. This is going to happen. It is an inside joke that twelve people really understand, and the rest of the TTRPG community on Twitter puts up with.
So, ask around. Ask me. For real. If I do not have a solid suggestion, I can boost the question at least, and get the word out to people who may be a bit more in the know with currently available systems (my knowledge is a bit archaic at this point).
I guarantee that there is a system out there that can suit your needs, or at least get you pretty close to where you want to go!