Two entries with the same name inside a ZIP? Detecting duplicate entries with Python zipfile

プログラミング・Web開発カテゴリを表すパンダのイラスト Programming / Web Development

About this articleThis article is created using an automated generation workflow leveraging generative AI. Based on the official specifications of Python's standard zipfile library, we outline the procedure for reproducing cases where duplicate names exist inside a ZIP using dummy data.

Verification status: 📘 Confirmed with Python official specifications – Actual execution of the provided code has not been verified

Two entries with the same name inside a ZIP? Detecting duplicate entries with Python zipfile

When you open a ZIP and look at the file list,report.csvit may look like there is only one. However, when automating processing, it can lead to reading unexpected contents or mismatched counts after extraction. One candidate cause is thepresence of multiple entries with the same name inside the ZIPstate. The names displayed in the ZIP list and the individual items held by the ZIP are not necessarily one-to-one.

Python'szipfile.ZipFile.infolist()returns individual items asZipInfo. On the other hand, aggregating them into a dictionary using only the name as a key loses duplicates. In this article, we will detect duplicates in a dummy ZIP created in memory without extracting to disk, and consider how to handle them in business operations.

Target environment and safe experimental scope

ItemCondition
TargetPython 3 standard libraryzipfile、io、collections
InputTwo dummy CSVs in memory. No actual files or personal information are used
ExperimentCreating ZIP files, listing contents, detecting duplicate names, and reading individual entries
External communicationNone. No additional packages or administrator privileges are required.
VerificationRefer to the official specifications. Execution of the code provided below has not been verified.

The target of the operation isBytesIOwhich is a byte sequence in memory. It does not extract or overwrite actual ZIP files on your computer. Before applying this to business files, please first make a prediction for this small example.

First, let's try putting in two files with the same name.

In the following code, the samereport.csvTwo items with different contents are written to the ZIP under the name. Depending on the version of Python, when writing duplicate names,UserWarningis displayed. This is an observation target and does not need to be unconditionally ignored just to suppress the warning.

import io
import zipfile
from collections import Counter

memory = io.BytesIO()
with zipfile.ZipFile(memory, "w") as z:
    z.writestr("report.csv", "id,value\n1,old\n")
    z.writestr("report.csv", "id,value\n2,new\n")
    z.writestr("note.txt", "sample only")

memory.seek(0)
with zipfile.ZipFile(memory, "r") as z:
    entries = z.infolist()
    counts = Counter(item.filename for item in entries)
    print("entries:", len(entries))
    print("names:", [item.filename for item in entries])
    print("duplicates:", {name: n for name, n in counts.items() if n > 1})
    for item in entries:
        if item.filename == "report.csv":
            print("member:", z.read(item).decode("utf-8").strip())

Based on the specifications, the expected result is that the total number of entries is 3, the list of namesreport.csvappears twice, and the duplicate count is displayed as 2. In the final loop, since the twoZipInfoobjects are passed individually, you can verify both the old and new contents. The order displayed here is the order in which they were added during creation. However,you must not rely on the order of general ZIP files as a basis for old/new status or correctness.。

The important point is thatz.read(item)instead of a string, you passZipInfoto. When names are the same, simply writingz.read("report.csv")makes it ambiguous which entry is intended. The official documentation also states that when distinguishing duplicate names,ZipInfoexplains that can be passed.

Why duplicates remain in the list

A ZIP is not a "unique dictionary keyed by name," but an archive containing information for individual file items.infolist()returns a sequence containing not only name strings, but also metadata such as size, compression method, and CRC.ZipInfosequence. Even with the same name, they can exist as separate entries.

Visualizing this difference in a single diagram makes it clear where information is lost.

flowchart TD
    A["ZIP内の全エントリー"] --> B["infolistで個別のZipInfoを取得"]
    B --> C["filenameごとに件数を集計"]
    C --> D{"同じ名前が複数あるか"}
    D -->|はい| E["重複を報告し個別に確認"]
    D -->|いいえ| F["次の検査へ進む"]

The key point in the diagram is tocount the number of items before grouping them by nameIf you overwrite and assign values to a dictionary keyed by filename, only the last remaining value may be visible. Even if you check for duplicates using that dictionary, you cannot recover how many entries were originally in the ZIP.

Countermaintains the occurrence count for each item name. While the process of finding duplicates is simple, operationally it is necessary to separate "detecting duplicates" from "deciding which one to adopt." This is because the latter must be determined by the archive creator, the delivery specification, and the data version control rules.

What happens if we change just one condition?

Try changing the secondwritestr("report.csv", ...)towritestr("report-new.csv", ...). Only the name is changed. The expected duplicate dictionary becomes empty, and the total entry count remains 3. This comparison shows that what is being inspected is not whether the file contents match, but rathername collisions.

Conversely, cases where names differ but the contents are identical are not judged as duplicates by this inspection. If you want to check for content duplication, you need to separately compare hashes or record identifiers. Additionally,Report.csvandreport.csvare treated as different names in Python string comparison. Because the conditions differ from name collisions when extracting onto Windows, additional verification tailored to the destination file system is required.

Another boundary condition isdata/report.csvandreport.csvis. Since they have different paths inside the ZIP, they will not be duplicated in thisfilenamecomparison. However, if you ignore the folder structure during extraction and save them solely by filename, a collision will occur. Define what constitutes the "same name" in conjunction with the actual subsequent processing.

What to Isolate When Failures Occur?

zipfile.BadZipFileIf an error occurs, do not immediately assume it is a duplication issue. The ZIP structure itself may be corrupted.is_zipfile()While there are ways to pre-check the format using, a successful format check does not guarantee that all data can be read properly. Exception handling and validation during read operations are required.

Even if the filename can be read, the character encoding of the content is not necessarily UTF-8. In the experiment, I wrote ASCII strings equivalent to UTF-8 myself, sodecode("utf-8")is used, but for enterprise CSVs, character encoding must be verified separately. If decryption fails, isolate it as a character encoding issue of the target file rather than a ZIP duplication check.

If the ZIP contains encrypted items, they may be unreadable without a password. Furthermore, it is dangerous to load ZIPs with huge extraction sizes or numerous entries without restriction. The code provided here is instructional material for small fictitious data and is not production code implementing capacity limits, count limits, and timeouts required to safely process unknown archives.

Additionally, consider the possibility that filenames might contain../or strings that look like absolute paths. Avoid implementations that concatenate and write names inside the ZIP directly to arbitrary output destinations. Since the current method isinspection without extraction, there is no need to write to disk before evaluating path safety.

For Enterprise Use: Detecting and Stopping Duplications

For example, if the contract states that a ZIP received every morning from a partner company should containsales.csvprecisely one, you first retrieve the list of archive entries andsales.csvconfirm that there is exactly 1 instance. If there are 0, it is missing; if there are 2 or more, it is treated as a duplicate and acceptance is put on hold. Rather than automatically selecting either one, requesting a resend from the partner company makes it easier to track down the cause of aggregation errors.

Acceptance inspection is not complete with just the filename. Verify expected CSV columns, record counts, dates, character encoding, uncompressed and compressed sizes, and checksums if necessary. This is because the absence of duplication is merely one necessary condition and not proof of correct content. Especially in processes where importing the same file twice causes issues, such as payroll or billing, inspect internal archive duplications separately from the resending of previously accepted data.

In operational logs, recording the reception date and time, archive identifier, entry count, duplicate names, and the reason for retention will aid in investigations. However, if the actual filenames contain customer or personal names, evaluate the log's publication scope and retention period. There is no need to dump all files within the ZIP into the log.

The criteria for adopting this method are clear.If the business contract requires names within the ZIP to be unique, reject duplicates before ingestion. On the other hand, if handling special archives that intentionally contain items with the same name, process them byZipInfo rather than by name, and clearly define the selection rules. It is important not to assume a ZIP is corrupted simply because duplicates are found.

Official Information and Verification Scope

Expected results are explained based on the specifications in official documentation. Execution results of the provided code, extraction displays on specific OS environments, and Mermaid rendering in WordPress have not been empirically verified. If results differ in your environment, record and compare the Python version, ZIP creator, character encoding, and exception messages.

Document information

Article title
Two entries with the same name inside a ZIP? Detecting duplicate entries with Python zipfile
Published
Updated
Source
https://papanda925.com/?p=18601&lang=en

License: Text and original figures for which this site holds the relevant rights are available under CC BY 4.0 , unless otherwise noted. This article may include content created or edited with generative AI. If code has a separate license notice or a linked GitHub repository license, that license takes precedence for the code. Quotations, third-party materials, images, and trademarks are excluded from this license. Usage policy

Copied title and URL