A Deeper Look at The Exceptions to Ionization Energy and Electron Affinity

To understand the reason behind the exceptions to the periodic trends for ionization energy and electron affinity you need to understand stabilty and its relationship to energy.

Stability

Stability is a state of low energy. As a simple example, we know that a large round rock is more stable at the bottom of a hill than at that top If it is at the bottom, it will just sit there, at the top it could roll down. This matches the energy of the situation. At the top of the hill, the rock has more potential energy (is more unstable). At the bottom of the hill, the rock has less potential energy (is more stable).

 This means that when an atom is stable it has a lower energy than an unstable atom.

If a stable atom becomes unstable, it must absorb that energy from somewhere.

If an unstable atom becomes stable, it must release that energy.

Ionization Energy

When an electron is taken from a stable nitrogen atom, energy must be put it in. However the process takes more energy than expected, because you must put in the energy to remove the electron from the hold of the nucleus, but you must also add the energy associated with the decrease in stability. Thus the ionization energy of nitrogen is higher than you might expect.
When an electron is taken from an unstable oxygen atom, the electron configuration created (p⁴-->p³). This increase in stability causes a release of energy. So, the process of removing an electron from the pull of the nucleus costs energy, but we get a small amount of energy back. This is essentially a "rebate" making the total energy cost less.
As a result of these two things (nitrogen's ionization energy being higher than expected and oxygen's being lower they "trade places" in terms of ionization energy.

Electron Affinity

When an electron is added to an atom, energy is released as the potential energy of the system decreases.

When an electron is added to carbon (p²) it creates stability (p³). As a result the atom releases additional energy, making the total amount liberated greater.

When an electron is added to nitrogen (p³) it reduces stabilty (p⁴). As a result, the atom keeps some of the energy we would expect it to give off, decreasing the amount of energy liberated.

The most extreme version of this is when an electron is added to a noble gas. In this case, not only does the electron make the atom less stable, but the electron is added to the next energy level and therefore releases significantly less energy. In most cases this means that it actually requires an input of energy to make a nobel gas atom take an electron.

Another note about Electron Affinity

Since electron affinity is defined as the energy released when an atom gains an electron, we should remember that the release of energy is an exothermic process and that exothermic processes have negative energy values.
That means that when we say that the electron affinity goes up (as you move to the right of the periodic table), we mean that the amount of energy given off is larger. Numerically, that means that the number becomes more negative.