The weights don't need to add to anything.
They should be symmetric along the diagonal of perfect agreement. For example If observer A says small and B says medium, that should count the same as if A says medium and B says small. But it need not be the same weight as if A says medium and B says large, even if both are 1 category apart.
The symmetry means you only have to fill out your matrix above (or below) the diagonal.
Tom