Validation Rule Action allows apply complex validation rules. User can configure rules and when they are met system fill through exception message. 

If the rule is a part of the Action Group and Validation fails - other actions in the group will not be executed. 

• Add an action with the type = Rule (1).

See more info about creating Actions in Working with Actions article. 

• Enter Error message (2):

This text will be shown to the user if conditions entered are true. 

• Enter rules for Validation rule to fire (3).

Select the type: Query, Formula or Apex Call.
Each type allows to specify rules in specific way: see more details on working with Conditions in Working with Conditions chapter.