In the books, casting spells without words is a skill introduced in the 6th book that requires more effort than casting it normally. The movie just glossed over this.