Holding down the shift key while clicking the delete icon or pressing the delete key will cause items you are deleting to be immediately deleted, instead of being moved to the 'recycle bin'.