Finding Duplicate Parameters in URL Queries with Python: Safely Checking Empty Values and Percent-Encoding

About This Article
This article was created using an automated generation workflow leveraging generative AI. Based on the official Python 3.13urllib.parsedocumentation, we created a sample for inspecting offline URL queries. We verified 18 unit tests, syntax checks, and normal and abnormal CLI outputs on Linux / Python 3.13.5.

Verification Status: 🧪 Locally executed on Linux / Python 3.13.5 with 18 successful tests. Windows and macOS not verified. GitHub Actions and Codex CLI not executed.

Supported Environments, Capabilities, and Limitations

"The same item appears twice at the end of the URL, but which one is used?" When investigating the behavior of web applications and APIs, you encounter questions like this. For example, in the following string,tagappears twice, andemptyappears once.

tag=one&tag=two&empty=

The important point here is thatidentifying duplicatesanddetermining which value the server acceptsare distinct matters. This time, using the Python standard library,urllib.parse.parse_qsl()we count key-value pairs without losing them. This is a small test program that does not communicate with actual web services and does not display key names or values on the screen.

Inspecting pairs before converting them into a dictionary

parse_qsl()converts the query string into a list of key-value pairs. Meanwhile,parse_qs()returns a dictionary grouping values by key. Both have their uses, butif you want to check the original appearance order and duplicate count,parse_qsl()is easier to understand.

Official documentation is available in thePython 3.13urllib.parsedocumentation.keep_blank_valuesdetermines whether to retain empty values, andstrict_parsingis an argument that determines how to handle unparseable items. Since Python 3.10,separator='&'can be explicitly specified, and this tutorial also&uses only as a separator.

Prerequisites

Python 3.10 or later andaudit_query.py、test_audit_query.py and a README are used. Administrator privileges or additional libraries are not required. Actual measurements for this implementation were performed on Linux with Python 3.13.5; individual execution on Windows and macOS has not been verified.

The complete code and tests can be found in theDaily Code Samples section at the end. First, place the three files in a new working directory. Select the appropriate Python execution command for your environment, such aspy -3 on Windows orpython3 on Linux/macOS.

Caution: Do not pass actual authentication URLs or tokens to--query, as this sample does not include values in the output. However, command-line arguments themselves may remain in shell history or OS process lists. All strings below are fictitious.

Running the Code: An Example with Duplicate Keys

Run the following in the sample directory:

python3 audit_query.py --query 'tag=one&tag=two&empty='

The actual output verified on Linux with Python 3.13.5 is as follows. The exit code was0.

STATUS=VALID FIELDS=3 UNIQUE_KEYS=2 DUPLICATE_KEYS=1 EXTRA_OCCURRENCES=1 BLANK_VALUES=1

Breaking down the meaning of the numbers,FIELDS=3 indicates that there are 3 key-value pairs, andUNIQUE_KEYS=2 indicates that there are 2 distinct key names.DUPLICATE_KEYS=1 means there is 1 key name that appears two or more times.EXTRA_OCCURRENCES=1 is the total number of redundant occurrences of each key, excluding its first appearance.BLANK_VALUES=1indicates that there is one pair with an empty string value.

For exampletag=one&tag=two&tag=threethen even if there is only one type of duplicate key name, there are two extra occurrences. Separating these two ensures that you do not confuse 'how many types of keys have issues' with 'how many total extra items exist.'

Mechanism: What is Being Specified and Parsed

The core of the complete version is as follows. This is anexplanatory snippet, and the executable finished version including error handling and input limits uses the GitHub repository as the source of truth.

from urllib.parse import parse_qsl

pairs = parse_qsl(
    'tag=one&tag=two&empty=',
    keep_blank_values=True,
    strict_parsing=True,
    encoding='utf-8',
    errors='strict',
    max_num_fields=100,
    separator='&',
)
print(len(pairs))  # 3

keep_blank_values=TrueWithout it, items likeempty=might be excluded in standard parsing. In business input validation, whether an 'item was missing' or 'an item was present but empty' can have different meanings. Therefore, empty values are retained and counted.

errors='strict'specifies that if the byte sequence restored from percent-encoding is invalid as UTF-8, it results in an error instead of silently continuing with replacement characters. Also,max_num_fields=100is an educational limit to suppress parsing of extremely large numbers of items, and the URL specification itself does not mandate a 100-item limit.

Changing one condition: broken % encoding

Next is an example where the%is not followed by two hexadecimal digits.

python3 audit_query.py --query 'a=%ZZ'

The actual output is as follows, and the exit code was2.

STATUS=INVALID ISSUE=INVALID_PERCENT_ESCAPE

This is a surprising pitfall. Python'sparse_qsl()is not a validator that always errors on broken percent escapes. Therefore, the completed version independently checks whether%is followed by two hexadecimal digits prior to parsing.%FFIf it is formally two digits but invalid as UTF-8, such asINVALID_UTF8as.

This distinction allows you toincorrect encoding syntaxandthe inability to read the encoded byte sequence as UTF-8to be separated. It performs no repair or automatic retransmission, serving as educational material to check the cause before changing the input.

+、%2B, where results change depending on uppercase and lowercase

In URL queries,+is treated as a space, and%2Bbecomes a literal plus sign. Also,a=1&%61=2is a form where the sameakey appears twice after decoding. Key name case sensitivity is distinguished in this code. In other words,Tag=1&tag=2are counted as two distinct types of keys.

However,whether actual web applications distinguish key name case and whether they use the first, last, or all of duplicate values are separate issues. Please check the specifications on the server, framework, and application sides. This program does not guess those adoption rules.

Troubleshooting upon failure

returns exit code2if the input is invalid, and0if valid. This prevents displaying success on test failure and passing the data to the subsequent process. Furthermore,--queryCLI argument errors, such as whenargparseitself is not specified, are treated as2, and they must be distinguished from input validation in the program core even if they use the same

What was verified in the 18 tests

The executed command is as follows.

python3 -m unittest discover -p 'test_*.py' -v
python3 -m py_compile audit_query.py test_audit_query.py

We verified that all 18 tests passed and syntax checks passed on Linux / Python 3.13.5.We tested for duplicate keys, empty values,%61anda、+and%2B, invalid UTF-8, field count and character count limits, as well as CLI normal and error cases. We also confirmed through unit tests that the output does not contain dummy values. Codex CLI does not exist in this environment and was not executed. GitHub Actions was also not executed due to billing limits. Results for Windows/macOS have not been inferred or supplemented.

Do not confuse this with "safe URL" determination in practical applications

This tutorial only examines queryformat and occurrence countIt does not verify authentication, authorization, signatures, URL security, or correct server-side interpretation. Query strings may contain personal information or credentials. If you extend this into a mechanism for inspecting production data, you must separately design secure input channels that do not pass secrets via the command line, log handling, access permissions, and data retention periods.

Also, please refer to the security considerations regardingURL parsing in PythonDo not assume that the parsing results of the standard library are inherently safe input. This sample does not detect all types of malicious URLs.

Summary: Inspect duplicate keys by first trying a small

parse_qsl()retain key-value pairs, and count them while leaving empty values. Distinguish format errors by exit codes, and do not display input values. Even with just these three points, it is easier to organize the causes than simply viewing URLs as strings. Start by running it with three dummy items, and try changing one condition at a time, such as increasingtagto three times, to observe the differences in counting.

GitHub Samples and Official Information

• Daily Code Samples: query-parameter-audit (README, complete code, tests)

• Python 3.13 Official Documentation: urllib.parse

• Python 3.13 Official Documentation: argparse

• Python 3.13 Official Documentation: unittest