This writeup is meant to somewhat cover various pitfalls of grid based tactics games, and later with an understanding of them, explain why hexes are always less fun then squares.
CHAPTER 1: The Definition of Tactics
What sets tactical games apart from straight up turn based jrpgs, is movement & positioning. Thinking of a game where your location is meaningless or you cant select one, that's effectively just a normal turn based rpg. This applies even if you do have movement but it's very minor like rows or backline units, such as Final Fantasy I~VI, Etrian Odyssey, or Radiant Historia.
In other words the most important, most critical, most defining trait of a tactics game, is the player moving pieces on the board. Location, Movement, & Positioning are everything. You can directly think of tactical games on almost an extremely childish scale of 0 to 100. The more the player is moving and positioning their pieces on the board, the more it's a tactical game. The less the player is doing this, the less it's a tactical game. When a person wants to play a tactics game, their really saying "i want to play a turn based game where my positioning matters". Tactics games must emphasize this or players will be lost to other games.
CHAPTER 2: Pitfalls
1: Mixed turn order vs team phases.
In tactics games, pacing is a big issue. many are slow as fuck, and not a single tactics series has survived long-term trying to use mixed turns instead of phases. Mixed turns are when there is a global turn order, mixing ally and enemy units so some of yours, then some of the enemys, then some more of yours, etc. Phases has all allys take their turn, and then all enemys take their turn. To date, the largest tactical series in existence, is FE(Fire Emblem), SRW(Super Robot Wars), and Wars(Famicom Wars -> Super Famicom Wars -> Advance Wars -> Still Advance wars on NDS -> Hiatus -> Wargroove), and Xcom(Its oh hiatus now, but it has like 20 games). Everything else like FF tactics, Shining Force, Department Heaven, Ogre Battle, Front Mission, have all has their series long since end (or go to another genre).
When gameplay uses phases, not only can you plan out your entire team speeding up gameplay, and think more critically about everyone's movement, but you can even plan out the enemy's movement! This greatly increases the amount of units whos movement your considering all at the same time, to literally everything on the entire board making things feel far more tactical. Getting control over enemy movement is something of an "Ultimate" tactical experience, giving you complete understanding of where your pieces can go if you think hard enough. It also opens up the game to have to option to let you skip enemy phase, further speeding up the game! Altogether, it vastly increases the tactical feelings by increasing how many units movement you think of, while simultaneously speeding up the game.
Without phases, The game balance rather then on positioning or movement, focuses more on instant kills or 1-2 combos to stop units, and the gameplay starts devolving from there. Its to easy for enemys to get a lot of turns in a row and kill someone, so units get a revive spell or item. And from there, your playing much closer to a rpg where you nolonger care about movement anymore, and focusing more on revive timing and using human wave tactics. Or that awful thing where the player can slow down their units speed rating to perfectly intermix turns such that they can always drop a heal after every enemy attack (as that is both annoying to plan, and always gamebreaking).
2: attack ranges - big vs small.
Positioning matters, unless lots of units have 3~7+ attack ranges. Look at games like FF Tactics. You often dont care about your location, and as a direct result, the games are far less tactical. SRW can get away with it because the challenging part is the bosses where you want to use your 1-2 range attacks, and its the largest crossover series in the world (several times larger then super smash brothers), You can feel the sense of tactics slipping. When all non-melee attacks hit huge ranges, everyone's exact location stops mattering. Even xcom has it really only matter for discovering pods or if units have line of sight, rarely does anything else matter. (and even then, xcom has line of sight ignoring options like grenades, rockets, etc. If you watch high level xcom videos, by mid-game they often don't even stand in cover and abuse overwatch.)
SRW realized the magnitude of issue this created, and added support attacker that require adjacency, as well as aura effects after the first few games. Altho SRW not having walls in its stages will always be a massive negative, these additions did bring back position via adjacency bonuses. Later games also added adjacent relationship bonuses, and other adjacency bonuses, even eventually attacks you can only do with specific characters next to you. Unfortunatly, it also added support defender, taking a hit for a adjacent ally as a plentiful skill. Plenty of games like Yggdra Union, or Hoshagami rising blue earth just completely died out with attack ranges all being to large globally. Yggdra not only intermixes turns(kinda), but lets any allys in a large nearby area join in when you attack, later game those joiners themself being able to have their own nearby allys join as well, extending your armys attack range to literally anywhere that anyone on your entire team is attacking from. Meanwhile Hoshagami lets anyone use basically any spell in the game (you can see how that stops being tactical real fast). FF tactics and many other big range tactics games all do the same thing. Game devs want units to have different hp, The big ranges mean your squishies are easy kills, the player cant hide them, the dev adds support defend, or revive, and now almost all positioning (tactics) are dead.
3: Gridless (gridless movement, or gridless ability targeting).
Gridless means the cursor or targeting cone can be on any pixel, not that the grid exists but is invisible. For anyone that has actually played xcom 1, aiming a grenade is a great example of a terrible experience. The player devolves to pixel hunting to get the perfect AOE ability off. Same for smoke bombs, and literally every fucking AOE in xcom. This is amplified by flying enemys and the AOEs being 3D spheres, but even if we remove the 3rd axis and flatten our aim, the same problem still happens such as in games like phantom brave. For a non tactics example, look at the neptunia games. Its annoying to play games that do this! Do not design a game that makes players pixel hunt! Do not think you have an exception! any gridless mechanic in a tactics games is bad, period! There are other issues, but i feel like anyone who actually plays a game with this shit instantly understands why its bad, so ill move on.
4: Isometric
Being able to see shit is really important. Isometric games like FF Tactics or Ogre Battle add significant dev time for sprite rotations, camera rotations, and being able to see from atleast one angle. Whats less obvious, is again, how it hinders positioning vs map design. this problem manifests differently if the camera can turn or not so il separate these topics
Camera-less isometric games like FF tactics advance have a problem with seeing the player. if the map wants to have walls, for a unit to be on the other side, they cant be seen. This necessitates creating maps where everything has different heights instead, like fft advance or gungunir. However, these do not prevent units from attacking, rather they introduce height advantage mechanics. assuming we removed them, you still cant have maps where your on the other side of a wall, at most your ontop of a wall. But height advantage mechanics are problematic to, their un-fun to think of, and not a single tactical game exists were a player knows exactly how much of a bonus they get from one height vs another height. Theres just no good way to have proper walls, as you wouldn't be able to see characters on the other side, and this affects so much of the game. especially if we think about adjacency bonuses or grouping characters.
Camera-rotation isometric games still have a problem. the player prefers not having to interrupt their gameplay with moving the camera all the time, and any real complex indoor fortress map with lots of walls would be impossible to see from EVERY angle. Both camera types are also strongly against having large maps, as the height differences vs camera are suddenly a big issue. Also, yes, thats why every map in FF tactics A1 & A2 is small as fuck.
5: Multiturn (xcom, battle brothers)
In some games, units get multiple actions per units turn, and this is basically always a bad thing. It sounds good at first, its flexible, players can double move to speedup gameplay or double attack a enemy, maybe even tripple attack with some specific skill. It also completely destroys game balance in an extremely un-fun way. Really, just save this as a specific unit skill, or let 1 ally have the ability to give another ally their turn, don't make everyone have multi-turn!!!
The moment units can attack multiple times a turn, either their damage must be way lower, characters dying is normal, or accuracy ratings get much lower. Often all 3 at once because of the reality of multiple units targeting a single opponent. If 2 units target one target, they need to survive upto 4 hits. lowering acc helps, but both doesn't solve and annoys players. just letting people die a lot changes the meta so much it somewhat stops being tactical as positioning stops mattering. The damage can become abysmal, but that both defeats the purpose and makes the game a drawn out slog vs one attack that deals double damage. In xcom, there is gun ammo, and people are meant to die a lot. But often, noone actually beats or finishes xcom unless they savescum or cheat (and even then...). its literally a meme how often people start and quit within the first 25% of an xcom campaign. It feels bad! I'd love to talk battle brothers where anyone can die instantly because of mixing multiturn with hex, but i'm really trying to save hex for last.
6: Zone of Control
ZoC is a game mechanic tried out in lots of old bad tactics games. In a game with ZoC, units cannot walk adjacently past enemy units. Any unit while moving that becomes adjacent to an enemy CANNOT move any further. This mechanic is bad because it reduces the amount positioning matters, as you don't even need to create a wall of units to stop enemy movement, 1 tank is enough for a 3 tile choke point, just put him in the middle and no one can walk past! It also reduces the players ability to move their own units, and as we defined tactics and movement, were adding a mechanic to a tactics game, that makes it stop being a tactics game. Now you just have your frontline units, and backline, and same for the enemy. Noone can walk into anyone elses formation or backline, period.
ZoC found success most as a soft-implemented skill version. For example, in some games a unit can get a skill to attack a enemy that walks next to you and stop it's movement. Mostly however, skills that restrict the other teams movement and combine it with great damage like a reaction attack are broken, and no such games that have them still survived.
CHAPTER 3: HEXES ARE ALWAYS BAD
Generally, humans think of spaces in grids. we have grid paper, grid menus, even reddit itself where we are talking has grid and list modes. Hex is something most people have a hard time thinking about, but its worse then that. Coincidentally, i recently also opened a gundam G-gen game on nds and saw it was hex grid and the grid was invisible. Instead of suffering through a game with no english translation with hex grid, i closed the fucking game.
Hexes have a few major problems. First, the control scheme. Most players are going to play games with a controller, and for games with precisely defined locations like tactical spaces, Tactical players prefer the D-pad, and the D-Pad experience on hex is complete garbage. The D-pad only has 4 directions. Nomatter what, one of the directions wont go quite that direction when pressed. In hex, either the updown axis can be all adjacent, or the left right. If we assume the up down axis is all adjacent, then theres a top left and bottom left but no left. Pressing left, wont move the cursor left, but to one and only one of these. worse yet, then what? if it went top left, will it go top left again, or bottom left to try and keep the same height you started? if its always top left, you always have to press down, an extra input, to go where you want. if bottom left, the player must constantly remember their cursors parity state. These are both annoying and add friction to cursor movement, along with extra inputs.
Hex tactics has major issues with positioning and game balance. The most obvious one, hex has an imperfect wall problem. As said before, the part the D-Pad works well with can either be vertical or horizontal. I'll refer to the one it doesn't work well with, as the squiggly line. Players can't make effective walls on a squiggly, and the math on it is insane. Enemys might not get 2 actions, but if so much extra damage is incoming, it's not that different.
In a standard square grid unit wall, a unit infront of you can hit you, the ranged unit behind it, and on occasion, the ranged unit next to the enemy as well for a total of 3 incoming attacks, but usually only 2. In hex, the "good" formation against a wall with 2 exposed allys, 1 ally must survive 2 1-Range attacks, and 5 2-range attacks for a total of 7 strikes, more then DOUBLE the incoming damage of square grid. If the squiggly wall is the other way where the middle guy is forward, he has 3 1-range enemys and upto 7 ranged attacks incoming for 10 total, now more then TRIPPLE the damage of a square grid! And thats assuming not a single units in the game can hit from 3 tiles away. That aside, game balance already doesn't exist just from how much more damage is incoming alone!
Hex games always have the worst possible map design.
Even just graphically, If a 4x4 building wants to exist, the edges must be all wavey and fucked up looking. But no, it goes much worse then that. If there were actual maps, more units can use ranged attacks against you in squiggly hallways or across squiggly rivers, of course, you could just remove all map design an use a completely flat plains for every battle, but no one would do something that boring...right? Just kidding, thats exactly what they all do. Their always giant empty fucking plains, usually with some very tiny height differences, and thats it. Some have bushes, and thats all. It's the most boring, uninteresting as possible maps you have played in any video game.
Lets look at a tactical nightmare, battle brothers.
Big attack ranges, partially isometric view, multiturn units and a hex based grid. Theres even mixed turns instead of team phases. Its basically everything ive said is bad all in one game.
Battle brothers maps are not maps, their all 50x50 giant empty fields. Theres no deployment phase, units are just shoved into battle. Characters have multiple actions and the fatigue mechanic does not enough prevent it. theres nowhere to hide, and enemies can hit you at decent ranges. worst of all its fucking hex tactics
Theres no maps, everything dies without a fighting change, nothing feels fair, the game is ugly, combat is slow, you never think about your positioning or the enemys other then on turn 1. Everything about this games design implodes on itself the the point streamers makeup their own goal and the game is a complete RNG-fest to survive even the first 10 random battles. Oh, plus ranged attacks, add in extra long range (upto 4 times) and multiturn, and a unit must survive upwards of 20 attacks in 1 round. Needless to say, your permanently dieing units are fucked. Theres no good play, you will die. the game isn't, and cant be designed for any real gameplay. lowering the accuracy of everything by 40% wasn't enough, 1RN rolls isnt enough, there is literally no design trick to make this okay.
The games a sandbox game. theres no wincon, you just play until you dont. despite this, you can still lose a campaign. you can run out of units and need to start anew. IN A SANDBOX GAME. This is because to make death matter, you need gold to buy units. you can run out, and boom, if you full wipe without backup units, game over. the game literally has so many overlapping bad designs it has to have no objective to even be playable, and the results of these bad designs are so impactful you can wipe in a fucking sandbox game. imagine if you could wipe in minecraft? like, what the fuck??????
If it didnt have multiturn, accuracy wouldnt need to be so low, and it wouldnt be as annoying to play. If it wasn't hex tactics, you could make walls, non wiggly ones, or have actual maps. If it wasn't…everything, it could be a good game. The difference between Fire Emblem and Battle Brothers is night and day.
Do not make hex tactics games. The end.
Special Notes
The Civ / 4x games, do use hex, but are not meant to be tactical. They are political, research tech trees, alliances, etc. The hex is on purpose, your kingdom is supposed to be easy to collapse. it speeds up competitive matches. Also, they use a mouse instead of controller, and they often care so little about unit movement their set to auto movement. 4X is not magically different, its just not a tactics game. i mean its in the name, 4x, not tactics. its a different genre.