Earlier this year, during a conversation I was having with the Scottish singer-songwriter and folk musician Alasdair Roberts, he came out with a striking confession. He told me that his song ‘The Yarn Unraveller’, on his 2009 EP The Wyrd Meme, was inspired by Elizabeth Stewart, a Scottish traveller and occasionally-recorded folk singer with a highly distinctive voice and a huge repertoire of remembered songs. ‘I was thinking about her being this authentic custodian of the folk tradition,’ he explained, ‘being representative of this real tradition, and me and people like me as somehow removed from it, and yet interested and wanting to work somehow with it. But also the fact that we weren’t the real deal. Well, maybe we are….’

And this from a man who has released several albums of beautifully wayward magical ballads himself; whose flat is stuffed with all manner of arcane recordings of traditional music and folk instrumentalism; and whose father himself was a semi-professional folk singer. So, musicians are still bothered by questions about their entitlement to sing folk songs. How does that square with the fact that folk is supposed to be the people’s music, indeed, a people’s music? Why is there always this sense that it’s hallowed ground, where only the true-hearted may tread?

Such contradictions, several years ago, struck me as being intrinsic to the enduring appeal and revulsion many people hold towards the idea of folk. So I set about exploring the evolution of folk in Britain. I’m no folkie, I should add: I’m a former editor of Wire magazine and spend much of my time listening to and writing about a wide range of musics, noisy, experimental, avant-garde, free-spirited, digital and transgressive. A decade ago I would not have given British folk much shrift. Now things have changed.

Where do you begin, though, this far down the line historically, in trying to track the course of British folk music and its discontents? In my mind were books like Peter Ackroyd’s Albion: The Origins of the English Imagination, or A. L. Morton’s The English Utopia, books which aren’t afraid to treat indigenous culture as a kind of dream of a people about its homeland. At the same time I wanted to follow how the notion of ‘folk’ as a distinct entity has developed from the Victorian and Edwardian collectors, via its successive regenerations courtesy of pastoral composers, Marxists, folk-rockers, psychedelic tribes and free festival-goers. Why, in short, does the word ‘folk’ get applied to such different creatures as The Copper Family, Vaughan Williams, Ewan MacColl, Nick Drake, Incredible String Band, Fairport Convention, Crass, Kate Bush?

I approached the subject crabwise, letting my own enthusiasms, intuitions and musical tastes shape the narrative, while detecting undercurrents that ran deep throughout a century. I began to realise that, for instance, although folk was meant to be the music of the people, none of it was really ‘pop’. That word comes from the Latin ‘populus’, the urban population that was weaned on mass spectacle and controlled civic life. A group like the The Kinks, for instance, are unquestionably British in sound and tone, but they’re also urban(e) and, well, pop. Folk, by contrast, is the music of the ‘Volk’, a word born of the Teutonic Wald, the wild wood where society was organised very differently, ad hoc, bottom-up and frequently savage.

And that was the second trail to follow: the all-important connection of music and the land. Peter Ackroyd writes, ‘The poetry of England is striated with the shade that ancient trees cast, in a canopy of protection and seclusion,’ and the same goes for the music. Folk songs were preserved in seclusion for centuries, as an unofficial vernacular culture, until certain antiquarians and archivists decided they were about to die out, and so went in search of the villagers and labourers in whose memories they were preserved. It is these jottings – often sketchily transcribed and subject to all kinds of contingencies – which form the backbone of our zealously guarded ‘folk tradition’. Francis Child, collator of the famous ten-volume Child Ballads, was a Boston-based academic who did all his collecting remotely, by sending out thousands of letters to community and parish leaders in England and Scotland, requesting any songs they could transcribe from their locals. Cecil Sharp came late to folk music after failing at composition; a chance hearing of a morris dance in 1899 turned his head, and a later encounter with a Somerset man humming ‘The Seeds of Love’ in his garden in 1903 convinced him to begin collecting in earnest. The history of folk music is littered with such epiphanic, obsessive amateurism.

Regard any LP or CD sleeve of English orchestral music by, say, Elgar, Vaughan Williams, Butterworth, Gurney, Warlock: rolling vales, lush green pastures, winding lanes, spires rising from treetops. And yet there’s a truth in this cliché. These composers were of the generation that witnessed the Great War – saw what could happen in an instant to long-preserved green fields and landscapes, and developed a determination to conserve it. Vaughan Williams, too, grew up precisely in the age of conservation, the formation of the National Trust by philanthropic Victorian socialists. One such, William Morris, is something of a starting point in understanding English folk music. Why? Firstly, his 1890 novel News from Nowhere, a proto-sci-fi time-travel novel, was peculiarly pertinent, as it deals with a trip 200 years into the future… into a medieval utopia, with no government, no money and a society of craftspeople. Reading it confirmed my feeling that the British imagination and its visionary spirit is most comfortable imagining the future in terms of an idealised past (see William Blake’s phrase, ‘the Nature of my Work is Visionary or Imaginative; it is an Endeavor to Restore what the Ancients calld the Golden Age’). Beyond that, though, I discovered that Morris’ political meetings in Hammersmith were attended by Gustav Holst and Ralph Vaughan Williams, drawn to his intoxicating mix of socialist rhetoric and grassroots love of art, craft and the ancient spirits of the countryside. Morris, of course, was effusively opposed to industrial ‘scrape’, polluting and effacing the landscape, demolishing historic buildings, and so on. The early music of Vaughan Williams and Holst is bound up in this fervent embrace of the British countryside, and Vaughan Williams himself was almost as prolific as Sharp, Percy Grainger and others in his rural folk-collecting trips. His contemporary John Ireland, meanwhile, was seduced by the mystic power of England’s prehistoric landscapes, seen through the prism of Arthur Machen’s fictions, which deal with uncanny pagan presences emerging from long burial. Ireland acknowledged Machen as an influence on pieces like Mai-Dun, Sarnia, The Scarlet Ceremonies and Legend, whose rhapsodic tone poem for piano and orchestra was triggered by a vision of ghostly children dancing at Chanctonbury Ring, West Sussex. Already, following the folk tracks, we find ourselves in territory outside any conventional folk canon and yet inextricably bound up with it.

Ireland was one of many composers who retreated to the countryside in order to kickstart and refertilise their own creative energies. Examples include Rutland Boughton’s residency at Glastonbury from 1914, where he set up the town’s first festival and premiered his successful operas on Arthurian themes; Peter Warlock’s two-year ‘lost weekend’ with Ernest Moeran at Eynsford, Kent, an anarchic, drunken, riotous composers’ retreat; and Fairport Convention’s summer of 1969 spent at Farley Chamberlayne, Hampshire, recuperating from a van crash that had killed one of their members and where they wrote the material for the great English electric folk breakthrough, Liege and Lief. Bassist Ashley Hutchings repeated the process for his first incarnation of Steeleye Span in early 1970, at Winterbourne Stoke, where, with Tim Hart, Maddy Prior, Gay and Terry Woods, he coated British balladry in an incredible finery of chiming guitars, earthen rhythms and electrified dulcimer.

At the same time Vashti Bunyan was undertaking her epic walk, with a Romany caravan, horse and dog, the entire length of Great Britain, from Kent to the Isle of Skye, ostensibly to join up with another folk-rock icon, Donovan, who had recently purchased a set of islands in order to establish a hippy alternative community. By the time she and her boyfriend arrived, Donovan had moved on, but out of the journey came one of the most celebrated records from that time, Just Another Diamond Day. The Incredible String Band lived in glorious mind-expanding seclusion in Scotland and later Wales, and even Led Zeppelin’s Robert Plant and Jimmy Page wrote the folk/Celtic myth-inflected IV album at a cottage in Bron-Y-Aur, North Wales. Groups such as Traffic, Heron and even John Martyn literally recorded outside in fields. Meanwhile, the folk clubs set up by the 1950s revivalists nurtured the 60s anti-authoritarian underground, with its pastoral psychedelic vision of a new society, which in turn led to the ‘free festival moment’ in Britain – Glastonbury, Stonehenge, Windsor, Phun City – all of which, for me, resurrects the Digger mentality of the English Civil War – the seventeenth-century group of nonconformists who believed in egalitarianism and a better balance between people and the earth. In turn, Winstanley’s project to reclaim a common treasury of the land for the people has recently been appropriated by that ‘modern antiquarian’, Julian Cope, and the deeply underground ‘heathen folk’ scene.

And then, of course, there is the majestic, deeply refined pastoral music of Nick Drake. He had been brought up in idyllic isolation in the Midlands countryside but wrote much of his music while living a solitary life in London, which gave a peculiar exiled charge to songs such ‘River Man’ and ‘Chime of the City Clock’.

That sense of exile from Eden is the key, most of all, to the progress of inherently British music in the twentieth century. Britain, I believe, has its own ‘blues’, whose source is twofold. First there was the mass displacement of agricultural labourers from the countryside to newly created industrial towns and cities during the Industrial Revolution. This flowed swiftly on the Inclosures Act of 1760 which essentially legalised an extensive land-grab of common ground. Secondly, there was the decimation of those who went to fight in the trenches of the Great War. Shirley and Dolly Collins’s Anthems In Eden, recorded in 1969, is haunted by both these occurrences. The suite is shot through with pagan dances and pastoral folk songs, but becomes overcast towards the end, as the maypole which previously formed the centrepiece of the village is replaced with a war memorial to commemorate the young men who never returned home. The inventive, chamber-folk arrangements and unison singing are augmented by the period instruments of David Munrow, Christopher Hogwood and others, lending the sound an appropriate whiff of the antique. But that sense of the lost estate, the exile from the golden-age garden, is of crucial significance to so many British artists, right up to the likes of Kate Bush, Talk Talk and David Sylvian in the 1980s and 90s.

But when did folk music acquire its terminally unfashionable status? In the 1950s the likes of Peter Kennedy and Ewan MacColl created popular BBC radio programmes especially to promote the songs of labourers and city listeners’ country cousins. By the late 60s Kenneth Williams affectionately lampooned folk’s yokel tendencies with his Rambling Syd Rumpo character on Round The Horne; more recently Bob Fleming of The Fast Show’s ‘Country Matters’ took a similar satirical line. But in the UK music press of the early 70s, folk was taken as seriously as rock, progressive, jazz and funk. The damage was really done in the mid-70s, when ‘folk’ songs now generally considered naff began gaining a high chart profile: Strawbs’ ‘Part of the Union’, Steeleye Span’s ‘Gaudete’ and ‘All Around My Hat’, The Wurzels’ ‘The Combine Harvester (Brand New Key)’ and Fiddlers’ Dram’s ‘Day Trip To Bangor’. These songs were untypical of the sound of British folk music, but have come to represent a signature sound of parochial triteness. I deduce from this that all attempts to market folk music as pop are doomed to fail – the modes are simply incompatible.

But this is a long way from Cecil Sharp and Vaughan Williams, cadging a few ballads off locals for the price of a pint. And yet I do see a vast, interconnected root system, much like the tangled, invisible knot of the British constitution itself; a ‘silver chain of sound/without a break’, as George Meredith wrote in the poem that inspired Vaughan Williams’ The Lark Ascending. The raw materials of British folk are there for anyone to appropriate and re-imagine; ancient ballads have wormed their way through this system down the ages, and Alasdair Roberts is only the latest to pluck them from the ground and sow their seeds anew.