An idiomatic rendering of the first sentence might be "A gemstone of such great size alone costs 80 sesterces." As loqu says, tanta agrees with gemma.
The adjective tantus,-a,-um suggests size (with countable nouns) or extent (with uncountable noun), usually with the positive connotation of largeness. However, it is technically relative and may, according to context, have the negative connotation of smallness. There's also the diminutive form tantulus,-a,-um which always means "so small, slight, insignificant".
It's only the adverb tantum that can mean "only" (the original idea being "to such a degree"), as in the second sentence. So...
Num anulus sine gemma viginti tantum sestertiis constat? means "Surely the ring without a gemstone does not cost only twenty sesterces?" Remember that num anticipates a negative answer.
Ex mala malo
bono malo uesci
quam ex bona malo
malo malo malo.