That they are heavy and very hard for a good reason.
While it might be tempting to build with a less demanding material - say swedes, for example, which are very plentiful in Britain (including the Lake District) and are much lighter and softer than rocks, we need to understand that they don't make nearly as strong a wall as rocks do, and eventually even swedish walls start to go mushy.
