Copyright Traps for Large Language Models 

 Proceedings of Machine Learning Research

 Volume 235 JMLR 
 DMLR 
 TMLR 
 MLOSS 
 FAQ 
 Submission Format 

 [ edit ]

 Copyright Traps for Large Language Models
 Matthieu Meeus, Igor Shilov, Manuel Faysse, Yves-Alexandre De Montjoye 
 Proceedings of the 41st International Conference on Machine Learning , PMLR 235:35296-35309, 2024.

 Abstract

 Questions of fair use of copyright-protected content to train Large Language Models (LLMs) are being actively debated. Document-level inference has been proposed as a new task: inferring from black-box access to the trained model whether a piece of content has been seen during training. SOTA methods however rely on naturally occurring memorization of (part of) the content. While very effective against models that memorize significantly, we hypothesize - and later confirm - that they will not work against models that do not naturally memorize, e.g. medium-size 1B models. We here propose to use copyright traps, the inclusion of fictitious entries in original content, to detect the use of copyrighted materials in LLMs with a focus on models where memorization does not naturally occur. We carefully design a randomized controlled experimental setup, inserting traps into original content (books) and train a 1.3B LLM from scratch. We first validate that the use of content in our target model would be undetectable using existing methods. We then show, contrary to intuition, that even medium-length trap sentences repeated a significant number of times (100) are not detectable using existing methods. However, we show that longer sequences repeated a large number of times can be reliably detected (AUC=0.75) and used as copyright traps. Beyond copyright applications, our findings contribute to the study of LLM memorization: the randomized controlled setup enables us to draw causal relationships between memorization and certain sequence properties such as repetition in model training data and perplexity.

 Cite this Paper

 BibTeX

 @InProceedings{pmlr-v235-meeus24a,
 title = {Copyright Traps for Large Language Models},
 author = {Meeus, Matthieu and Shilov, Igor and Faysse, Manuel and Montjoye, Yves-Alexandre De},
 booktitle = {Proceedings of the 41st International Conference on Machine Learning},
 pages = {35296--35309},
 year = {2024},
 editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix},
 volume = {235},
 series = {Proceedings of Machine Learning Research},
 month = {21--27 Jul},
 publisher = {PMLR},
 pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/meeus24a/meeus24a.pdf},
 url = {https://proceedings.mlr.press/v235/meeus24a.html},
 abstract = {Questions of fair use of copyright-protected content to train Large Language Models (LLMs) are being actively debated. Document-level inference has been proposed as a new task: inferring from black-box access to the trained model whether a piece of content has been seen during training. SOTA methods however rely on naturally occurring memorization of (part of) the content. While very effective against models that memorize significantly, we hypothesize - and later confirm - that they will not work against models that do not naturally memorize, e.g. medium-size 1B models. We here propose to use copyright traps, the inclusion of fictitious entries in original content, to detect the use of copyrighted materials in LLMs with a focus on models where memorization does not naturally occur. We carefully design a randomized controlled experimental setup, inserting traps into original content (books) and train a 1.3B LLM from scratch. We first validate that the use of content in our target model would be undetectable using existing methods. We then show, contrary to intuition, that even medium-length trap sentences repeated a significant number of times (100) are not detectable using existing methods. However, we show that longer sequences repeated a large number of times can be reliably detected (AUC=0.75) and used as copyright traps. Beyond copyright applications, our findings contribute to the study of LLM memorization: the randomized controlled setup enables us to draw causal relationships between memorization and certain sequence properties such as repetition in model training data and perplexity.}
}

 Copy to Clipboard 
 Download 

 Endnote

 %0 Conference Paper
%T Copyright Traps for Large Language Models
%A Matthieu Meeus
%A Igor Shilov
%A Manuel Faysse
%A Yves-Alexandre De Montjoye
%B Proceedings of the 41st International Conference on Machine Learning
%C Proceedings of Machine Learning Research
%D 2024
%E Ruslan Salakhutdinov
%E Zico Kolter
%E Katherine Heller
%E Adrian Weller
%E Nuria Oliver
%E Jonathan Scarlett
%E Felix Berkenkamp 
%F pmlr-v235-meeus24a
%I PMLR
%P 35296--35309
%U https://proceedings.mlr.press/v235/meeus24a.html
%V 235
%X Questions of fair use of copyright-protected content to train Large Language Models (LLMs) are being actively debated. Document-level inference has been proposed as a new task: inferring from black-box access to the trained model whether a piece of content has been seen during training. SOTA methods however rely on naturally occurring memorization of (part of) the content. While very effective against models that memorize significantly, we hypothesize - and later confirm - that they will not work against models that do not naturally memorize, e.g. medium-size 1B models. We here propose to use copyright traps, the inclusion of fictitious entries in original content, to detect the use of copyrighted materials in LLMs with a focus on models where memorization does not naturally occur. We carefully design a randomized controlled experimental setup, inserting traps into original content (books) and train a 1.3B LLM from scratch. We first validate that the use of content in our target model would be undetectable using existing methods. We then show, contrary to intuition, that even medium-length trap sentences repeated a significant number of times (100) are not detectable using existing methods. However, we show that longer sequences repeated a large number of times can be reliably detected (AUC=0.75) and used as copyright traps. Beyond copyright applications, our findings contribute to the study of LLM memorization: the randomized controlled setup enables us to draw causal relationships between memorization and certain sequence properties such as repetition in model training data and perplexity.

 Copy to Clipboard 
 Download 

 APA

 Meeus, M., Shilov, I., Faysse, M. & Montjoye, Y.D.. (2024). Copyright Traps for Large Language Models. Proceedings of the 41st International Conference on Machine Learning , in Proceedings of Machine Learning Research 235:35296-35309 Available from https://proceedings.mlr.press/v235/meeus24a.html.

 Copy to Clipboard 
 Download 

 Related Material

 Download PDF 

 OpenReview 

 This site last compiled Wed, 07 Oct 2026 11:41:07 +0000

 Github Account 

 Copyright © The authors and PMLR 2026.  MLResearchPress