Imports from the search engine: the collection of online resources is still growing

The German National Library (DNB) has been endeavouring to expand its collection of non-physical media works (internet publications) for some time – particularly in the area of open access publications. These publications are freely available in the internet – they can be read, saved, linked or printed at no charge. The first important step towards implementation was taken last year.

Importing BASE data

In August 2022, the DNB began importing metadata on open access publications from the search engine BASE (Bielefeld Academic Search Engine). BASE is one of the world's biggest search engines for scientific documents. The index encompasses over 300 million documents from more than 10,000 data providers. The full texts of around 60% of the documents indexed in BASE are in the public domain. As more than 16 million objects come from 575 German sources, many of the indexed objects also fall within the German National Library's collection mandate.

Supplementation rather than replacement

Collection through BASE does not replace direct deposits with the DNB but rather supplements them. The DNB thus collects the open access publication data available in BASE by means of agreed standards. Institutional open-access publications that do not meet the DNB's deposit criteria for direct collection can thus become part of the DNB's collection after all.

This step involves the import of the metadata into the catalogue; the objects themselves are integrated at a later stage. The transfer is effected on the basis of new, efficient and adaptable processes for the collection of online publications.

How can multiple imports be prevented?

The DNB's collection of non-physical media works already encompasses a large number of data suppliers. There is accordingly a risk of duplication. This is mitigated by the precise selection of data from specific institutions that have not yet deposited publications with the DNB. Moreover, the metadata of each object – e.g. its identification number or title – are checked on import in order to prevent duplication among the selected data suppliers. After all, objects could already have reached the German National Library from other sources.

How does digital music get to the German Music Archive?

CDs and vinyl records, sheet music, historic piano rolls, audio cassettes or shellac records: the German Music Archive (DMA) at the German National Library (DNB) houses one of the world's biggest musical collections. However, two-thirds of music sales are now generated by audio-streaming – thus expanding the DNB's collection mandate. The collection of digital music publications has already been part of the DNB's statutory collection mandate for more than 15 years. But how can the DMA collect as many digital music resources as possible? This topic was discussed by the DNB, DMA and representatives of the music industry at an online workshop held on 12 October 2022.

Non-physical music in the German Music Archive

The collection of non-physical media, referred to as "online publications", has been part of the DNB's collection mandate since 2006. On hearing the term "online publications", most people probably think of e-books, e-journals, digital audiobooks or websites. This is absolutely correct, but it is by no means all: musical works are also classified as online publications, i.e. as "non-physical music". But how does non-physical music get to the DMA?

Digital collections: the story so far

In general, the task of collecting, cataloguing and indexing online publications has to be automated due to the enormous volumes involved. Although non-physical media works do not occupy conventional shelving space, their content still has to find its way onto the DNB's servers. This requires suitable standard metadata formats in standard file formats as well as commonly used interfaces. The requirements for individual media groups vary considerably. Something that works for e-books may be unsuitable for music files – and vice versa.

Starting out with text products

Since the appropriate workflows and interfaces cannot be developed simultaneously, the DNB is initially focusing on non-physical text products. As of 2022, the DNB had collected more than 12 million online publications in the form of e-books, university publications, e-journals, e-articles – and also digital sheet music. As with all media archived at the DNB, visitors can usually view them in the reading rooms on the DNB's premises in Leipzig and Frankfurt am Main.

Now it's music's turn

Preparations for the collection of non-physical music collections have been under way for several years. The challenge is to find out which standards the music industry uses. The most important questions in this context are: how widespread is the international data exchange format DDEX, and in which version? Which other file formats are used? Following a detailed evaluation, work has now begun on implementing DDEX in close cooperation with the music industry. Without this cooperation, the DNB would be at risk of offering a standard that could not be used meaningfully if at all by the depositing organisations. The depositing organisations are usually labels and may also include distributors or musicians.

Solidarity with the music industry

In the Berlin-based digital distribution company for the label Morr Music, DNB has found a pilot partner that strongly supports the development of DDEX as a metadata transport format. The DMA organised a workshop with the aim of underpinning this step, communicating with the music landscape and offering initial insights into the DNB's deliberations regarding online music publications. This took place on 12 October 2022. The virtual event with representatives of the music industry included a lively discussion on metadata exchange formats.

Who took part in the workshop?

The active participants included staff from the German National Library and

• the DDEX secretary,
• the Bundesverband Musikindustrie (Federal Music Industry Association, BVMI),
• the Verband unabhängiger Musikunternehmen (Association of Independent Musicians and Music Companies, VUT),
• Phononet, service provider of the Bundesverband der Musikindustrie,
• the Gesellschaft zur Verwertung von Leistungsschutzrechten (Society for the Exploitation of Ancillary Copyrights, GVL),
• the Gesellschaft für musikalische Aufführungs- und mechanische Vervielfältigungsrechte (Society for Musical Performance and Mechanical Reproduction Rights, GEMA),
• the University of Leipzig and
• a representative of Berlin 3 Services and the Berlin Music Commission.

Attendee contributions consisted partly of their own presentations and partly of input into the concluding panel discussion. The DNB and DMA took the opportunity to speak about the milestones reached in the collection of online music publications as part of a workshop report. They also gave an insight into what will happen next and in which time frame.

Opportunities and limits of DDEX

Contributions by the GVL, GEMA, the DDEX secretary and the DNB gave the 80-strong expert public an understanding of how DDEX is structured, what it can do, who uses it to which extent for what – and also what its limits are and what difficulties can arise when using it. The participants addressed a wide range of topics, especially during the closing panel discussion. Thy expressed both ideas and concerns. One of the suggestions raised was whether it would be worthwhile setting up a lean version of DDEX especially for deliveries to libraries. This idea will be discussed further with other European national libraries in the near future.

Focusing on the concerns of small and mid-sized players

Other ideas included whether the DNB should access existing music databases maintained by GEMA, GVL or Phononet. Provided appropriate agreements were put in place, this could be another way in which the DNB could continue expanding its collection. These deliberations will continue in greater depth in the future. And which concerns were expressed? These were often based on the apprehension that the DDEX as a metadata standard could be too complex for small and mid-sized depositors. In certain circumstances, there could be a need for other options allowing small labels and artists without labels to deposit their music with the DNB directly.

Everything step by step

The DNB is in any case pursuing the plan of realising the deposit of online music publications step by step – and expanding the available deposit options still further. Concerns and requests expressed early on are therefore of great value. The course is now being set for the deposit channels of the future. In general, both the participating institutions and the expert public praised the DNB for its approach. Its commitment has created great transparency with regard to further proceedings. The fact that the DNB is already giving due consideration to the varying circumstances of depositors of all sizes has been positively received. This is the only way in which as many rights holders as possible can transfer their music – so that non-physical music finds its way into the DMA and is preserved for future generations. 

Are you interested in the workshop's content? The entire virtual event was recorded. You can watch the video and the speakers' presentations here.

12 million online publications

The German National Library has been collecting online publications since 2006. Online publications are media works published in the internet, e.g. e-journals, e-papers, e-books, digital audio books, websites and music files. Our task is to catalogue and archive them, guarantee their authenticity and ensure that they will remain available in the long term. The inventory of online works is increasing exponentially.

As of November 2022, the collection had grown from 10 million to 12 million online publications, an increase of 2 million in just 17 months. Nowadays, more than 60% of all the media works deposited every year are online publications. This increase clearly shows how rapidly the German National Library’s digital holdings are growing.