According to its advocates, Bayesian reasoning is a way of seeing the world, and our beliefs about the world, in the light of probability theory, in particular Bayes's Theorem or Bayes's Rule. This probability-theoretic way of seeing the world can apply to scientific issues, to tasks in machine learning, and to everyday life.
To start learning, visit Arbital's Guide to Bayes's Rule.
After that, consider visiting the Bayesian update page.