I can't think of any other evidence, in the Writtle book (Sacred Mounds, Holy Rings) there is a report on the seeds and fruit identified on the land surface beneath the mound, and the turf stack, over 1100 species, mostly very familar wild flowers of today. I think it was the vicar Smith, when Silbury was opened by the miners, he found what he thought was mistletoe laid on the stones around the original mound....
A practical builder would say, its easier to work in the dry summer months than wet winter months, so the argument could still stand. Notice Leary disses Dames and others because of the folklore mythology they have built up around Silbury - probably means Leary is bringing out a book with his theory ;)