Currently it is not possible to restrict access to ALL artifacts from a specific configuration for a group, while still having access to the state of the build (successful, failed, running, ...). As soon as a configuration is defined in a group's permission, it is both visible and available for download.
A separate permission is required which allows access control to artifacts to deny access for specific user groups.