10 Characters $neq$ 10 Bytes: Comparing UTF-8 and Shift_JIS Byte Counts in PowerShell

About This Article
This article reviews .NET’s Encoding API and the UTF-8 specification, organized around observing the differences between character counts and byte counts using PowerShell.

Verification Status: 📘 Official Specifications Confirmed, PowerShell Actual Environment Unverified
Since the availability of Shift_JIS-based code pages varies across PowerShell/.NET environments, handling for cases where they cannot be retrieved is included.

“10 characters” does not always mean “10 bytes.” Before strings are saved or transmitted, they are converted into a byte sequence by an Encoding, and that byte count is determined by the combination of the characters and the Encoding.

This difference becomes important when integrating with fixed-length data, CSVs, APIs, and legacy business systems.

Strings and Byte Sequences Are Different Things

flowchart LR
    A[文字列] --> B{Encoding}
    B -->|UTF-8| C[UTF-8 byte列]
    B -->|code page 932| D[Shift_JIS系 byte列]
    C --> E[byte数]
    D --> F[byte数]

In PowerShell/.NET, you can use GetByteCount to count the number of bytes generated by that Encoding.

Checking with UTF-8

$texts = @(
    'ABC',
    '日本',
    'A日本',
    '😀'
)

$utf8 = [System.Text.Encoding]::UTF8

foreach ($text in $texts) {
    [pscustomobject]@{
        Text       = $text
        StringSize = $text.Length
        UTF8Bytes  = $utf8.GetByteCount($text)
    }
}

Even for characters that “look like 1 single character,” the UTF-8 byte count varies between ASCII-range characters, Japanese characters, and emojis.

String.Length Is Not Always Equal to the “Number of Characters Seen by Humans”

.NET strings are treated as a sequence of UTF-16 code units. Therefore, for supplementary plane characters and the like, the visual character count and String.Length may not match.

flowchart TB
    A[見た目の文字] --> B[.NET String]
    B --> C[UTF-16 code units]
    B --> D[Encodingでbyte列へ]
    C --> E[String.Length]
    D --> F[GetByteCount]
    E -. 同じ概念ではない .- F

It is important not to oversimplify this into just two categories: “character count” and “byte count.”

Comparing with Shift_JIS Code Page 932

In PowerShell 7 / .NET, some environments require registering CodePagesEncodingProvider in order to use legacy code pages.

try {
    $sjis = [System.Text.Encoding]::GetEncoding(932)
}
catch {
    [System.Text.Encoding]::RegisterProvider(
        [System.Text.CodePagesEncodingProvider]::Instance
    )
    $sjis = [System.Text.Encoding]::GetEncoding(932)
}

$utf8 = [System.Text.Encoding]::UTF8

foreach ($text in @('ABC', '日本', 'A日本')) {
    [pscustomobject]@{
        Text       = $text
        UTF8Bytes  = $utf8.GetByteCount($text)
        SJISBytes  = $sjis.GetByteCount($text)
    }
}

Even for the same string, changing the Encoding changes the byte sequence.

Don’t Assume “Japanese Is 2 Bytes in Shift_JIS”

There are characters that are not included in the code page, as well as characters expressed in a single byte. Furthermore, treating Windows code page 932 and the name “Shift_JIS” as having the exact same specification loosely can lead to misunderstandings in the details. This is treated as an experiment to retrieve and compare code page 932 in PowerShell/.NET.

Why Issues Occur with Fixed-Length Data

In legacy system integrations, field widths are sometimes defined in bytes rather than “characters.”

Field A: 10 bytes
Field B: 20 bytes
Field C: 8 bytes

If you simply slice a string using Substring before saving, it may exceed the expected byte count.

flowchart LR
    A[入力文字列] --> B[文字数で切る]
    B --> C[Encoding]
    C --> D{指定byte幅以内?}
    D -->|No| E[固定長レコードがずれる]
    D -->|Yes| F[次の項目へ]

Conversely, carelessly cutting a byte sequence midway can potentially split multi-byte characters in half. With fixed-length data, you need to verify “which Encoding is used and how many bytes it takes” as part of the specification.

Consider BOM and Line Breaks Separately

Encoding.GetByteCount(string) calculates the number of bytes when that string is encoded.

On the other hand, the actual file size may include the following:

• BOM

• Line breaks such as CRLF / LF

• Delimiters

• File-wide headers

Do not confuse the “byte count of the string itself” with the “saved file size.”

Verifying with Actual Files

After saving a string, comparing the actual file size makes it easier to understand.

$text = '日本'

$path = '.utf8-sample.txt'
[System.IO.File]::WriteAllText($path, $text, [System.Text.UTF8Encoding]::new($false))

$bytes = [System.IO.File]::ReadAllBytes($path)

[pscustomobject]@{
    TextByteCount = [System.Text.Encoding]::UTF8.GetByteCount($text)
    FileByteCount = $bytes.Length
}

Here, BOM-less UTF-8 is explicitly specified, so you can compare while reducing extraneous conditions.

Summary

• Strings are converted into byte sequences via Encoding

• String.Length, visual character count, and byte count are not the same concepts

• UTF-8 and code page 932 result in different byte counts even for the same string

• Fixed-length data should be evaluated based on the specified Encoding and byte width, not character counts

• File sizes containing BOMs or line breaks should be considered separately from GetByteCount of the string itself

Official Resources & Primary Sources

Microsoft Learn — Encoding.GetByteCount
https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding.getbytecount

Microsoft Learn — Encoding.GetEncoding
https://learn.microsoft.com/en-us/dotnet/api/system.text.encoding.getencoding

Microsoft Learn — CodePagesEncodingProvider
https://learn.microsoft.com/en-us/dotnet/api/system.text.codepagesencodingprovider

RFC 3629 — UTF-8
https://www.rfc-editor.org/rfc/rfc3629.html