Generally, it is a bad idea to allow regular users on a linux server. OpenSSH provides directives to control who does and does not have access via ssh:
AllowUsers AllowGroups DenyUsers DenyGroups
Any one of those directives can be used. AllowUsers and AllowGroups is defined as “allow only the users or the users of the listed groups to have access to the system”. DenyUsers and DenyGroups is defined as “allow all users except those listed or allow all users except those who are in the listed groups”.