问题 修复由UTF-8和Windows-1252组成的文件

这是我写的原因之一 统一:: UTF8。使用Unicode :: UTF8，这在使用回退选项时是微不足道的 统一:: UTF8 :: decode_utf8（）。

use Unicode::UTF8 qw[decode_utf8];
use Encode        qw[decode];

print "UTF-8 mixed with Latin-1 (ISO-8859-1):\n";
for my $octets ("\xD0 \x92 \xD0\x92\n", "\xD0\x92\n") {
    no warnings 'utf8';
    printf "U+%v04X\n", decode_utf8($octets, sub { $_[0] });
}

print "\nUTF-8 mixed with CP-1252 (Windows-1252):\n";
for my $octets ("\xD0 \x92 \xD0\x92\n", "\xD0\x92\n") {
    no warnings 'utf8';
    printf "U+%v04X\n", decode_utf8($octets, sub { decode('CP-1252', $_[0]) });
}

输出：

UTF-8 mixed with Latin-1 (ISO-8859-1):
U+00D0.0020.0092.0020.0412.000A
U+0412.000A

UTF-8 mixed with CP-1252 (Windows-1252):
U+00D0.0020.2019.0020.0412.000A
U+0412.000A

Unicode :: UTF8是用C / XS编写的，只在遇到生成错误的UTF-8序列时调用回调/回退。

最近我遇到了严重混合的UTF-8，CP1252和UTF-8编码的文件，然后解释为CP1252，然后再次编码为UTF-8，再次解释为CP1252，依此类推。

我写了下面的代码，对我来说效果很好。它查找典型的UTF-8字节序列，即使某些字节不是UTF-8，也是等效CP1252字节的Unicode表示。

my %cp1252Encoding = (
# replacing the unicode code with the original CP1252 code
# see e.g. http://www.i18nqa.com/debug/table-iso8859-1-vs-windows-1252.html
"\x{20ac}" => "\x80",
"\x{201a}" => "\x82",
"\x{0192}" => "\x83",
"\x{201e}" => "\x84",
"\x{2026}" => "\x85",
"\x{2020}" => "\x86",
"\x{2021}" => "\x87",
"\x{02c6}" => "\x88",
"\x{2030}" => "\x89",
"\x{0160}" => "\x8a",
"\x{2039}" => "\x8b",
"\x{0152}" => "\x8c",
"\x{017d}" => "\x8e",

"\x{2018}" => "\x91",
"\x{2019}" => "\x92",
"\x{201c}" => "\x93",
"\x{201d}" => "\x94",
"\x{2022}" => "\x95",
"\x{2013}" => "\x96",
"\x{2014}" => "\x97",
"\x{02dc}" => "\x98",
"\x{2122}" => "\x99",
"\x{0161}" => "\x9a",
"\x{203a}" => "\x9b",
"\x{0153}" => "\x9c",
"\x{017e}" => "\x9e",
"\x{0178}" => "\x9f",
);
my $re = join "|", keys %cp1252Encoding;
$re = qr/$re/;
my %cp1252Decoding = reverse % cp1252Encoding;
my $cp1252Characters = join "|", keys %cp1252Decoding;

sub decodeUtf8
{
    my ($str) = @_;

    $str =~ s/$re/ $cp1252Encoding{$&} /eg;
    utf8::decode($str);
    return $str;
}

sub fixString
{
    my ($str) = @_;

    my $r = qr/[\x80-\xBF]|$re/;

    my $current;
    do {
        $current = $str;

        # If this matches, the string is likely double-encoded UTF-8. Try to decode
        $str =~ s/[\xF0-\xF7]$r$r$r|[\xE0-\xEF]$r$r|[\xC0-\xDF]$r/ decodeUtf8($&) /eg;

    } while ($str ne $current);

    # decodes any possible left-over cp1252 codes to Unicode
    $str =~ s/$cp1252Characters/ $cp1252Decoding{$&} /eg;
    return $str;
}

这与ikegami的答案有类似的限制，除了相同的限制也适用于UTF-8编码的字符串。