This paper aims at characterizing and improving the metrological performances of current and voltage instrument transformers (CTs and VTs) in harmonic measurements in the power system. A theoretical analysis is carried out to demonstrate that, due to the iron core nonlinearity, CT and VT output signal is distorted even when the input signal is a pure sine wave. Starting from this analysis, a new method for CT and VT characterization and compensation is proposed. In a first step, they are characterized in sinusoidal conditions and the harmonic phasors of the distorted output are measured; in the second step, these phasors are used to compensate the harmonic phasors measured in normal operating conditions, which are typically distorted. The proposed characterization and compensation techniques are called SINusoidal characterization for DIstortion COMPensation (SINDICOMP). Several experimental tests, using high-accuracy calibration setups, have been performed to verify the proposed methods. The experimental results showed that the SINDICOMP technique assures a significant improvement of CT and VT metrological performances in harmonic measurements.