The only downside to 6mm for rpg and skirmish gaming is that your characters won’t be as detailed and can be harder to tell apart. They could also lack tiny elements of detail that tell the character’s story. Terrain could also be an issue because you will probably lack the intricate detail of 28mm stuff, as sometimes the scale feels off if every character is stuck to a base that is basically the size of a normal sized bedroom and you are trying to fit a few characters in a small space.
I am going to pretend you never said that!
Though interestingly (or not?) that is why I oversize my buildings and vehicles so that when a based figure is placed next to them they look more in keeping/scale.