[Bf-extensions-cvs] SVN commit: /data/svn/bf-extensions [4257] trunk/py/scripts/addons/ io_scene_ms3d: fix/workaround for character encoding ('ascii' is used for now, it should be 'cp1252' finally, but there are issues with printing on system console a
Beta Tester
alpha-beta-release at gmx.net
Mon Feb 11 22:27:58 CET 2013
hi Campbell,
i think i have to use that encoding, when i want to encode/decode namings of ms3d in a correct
way, because ms3d natively use cp1252 as codepage (because it runs in a windows environment).
so on import time i decode ms3d strings to blender strings (utf-8) and on export i encode
blender strings back to ms3d compatible ones...
but not yet, i use "ascii" instead "cp1252", for now (ascii is a subset of cp1252), because when
i want to use cp1252, i get during windows system console output and later on export then i
encode and write back to file an "UnicodeError" exception.
https://projects.blender.org/tracker/index.php?func=detail&aid=34189
or am i misunderstanding the meaning of decode / encode?
bytes.decode('cp1252') brings a cp1252 encoded string to blenders utf-8.
and str.encode('cp1252') brings blender strings back to cp1252 ones...
is that correct... and when i use 'ascii' instead of 'cp1252' it does the same, but only this
more limited codepage?
----- Original Message -----
From: "Campbell Barton" <ideasman42 at gmail.com>
To: <bf-extensions-cvs at blender.org>
Sent: Monday, February 11, 2013 3:17 AM
Subject: Re: [Bf-extensions-cvs] SVN commit: /data/svn/bf-extensions [4257]
trunk/py/scripts/addons/ io_scene_ms3d: fix/workaround for character encoding ('ascii' is used
for now, it should be 'cp1252' finally, but there are issues with printing on system console and
struct.pack or io .FileIO.write)
> On Mon, Feb 11, 2013 at 3:46 AM, Alexander Nussbaumer
> <alpha-beta-release at gmx.net> wrote:
>> Revision: 4257
>>
>> http://projects.blender.org/scm/viewvc.php?view=rev&root=bf-extensions&revision=4257
>> Author: beta-tester
>> Date: 2013-02-10 16:46:54 +0000 (Sun, 10 Feb 2013)
>> Log Message:
>> -----------
>> fix/workaround for character encoding ('ascii' is used for now, it should be 'cp1252'
>> finally, but there are issues with printing on system console and struct.pack or
>> io.FileIO.write)
>>
>> Modified Paths:
>> --------------
>> trunk/py/scripts/addons/io_scene_ms3d/__init__.py
>> trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py
>>
>> Modified: trunk/py/scripts/addons/io_scene_ms3d/__init__.py
>> ===================================================================
>> --- trunk/py/scripts/addons/io_scene_ms3d/__init__.py 2013-02-10 14:51:42 UTC (rev 4256)
>> +++ trunk/py/scripts/addons/io_scene_ms3d/__init__.py 2013-02-10 16:46:54 UTC (rev 4257)
>> @@ -23,7 +23,7 @@
>> 'description': "Import / Export MilkShape3D MS3D files"\
>> " (conform with MilkShape3D v1.8.4)",
>> 'author': "Alexander Nussbaumer",
>> - 'version': (0, 95, 1),
>> + 'version': (0, 95, 2),
>> 'blender': (2, 65, 3),
>> 'location': "File > Import & File > Export",
>> 'warning': "",
>>
>> Modified: trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py
>> ===================================================================
>> --- trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py 2013-02-10 14:51:42 UTC (rev 4256)
>> +++ trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py 2013-02-10 16:46:54 UTC (rev 4257)
>> @@ -37,6 +37,9 @@
>> from sys import (
>> exc_info,
>> )
>> +from codecs import (
>> + register_error,
>> + )
>>
>> ###############################################################################
>> #
>> @@ -97,6 +100,22 @@
>> HEADER = "MS3D000000"
>>
>>
>> + ## TEST_STR = 'START@€@µ@²@³@©@®@¶@ÿ@A at END.bmp'
>> + ## TEST_RAW = b'START@\x80@\xb5@\xb2@\xb3@\xa9@\xae@\xb6@\xff at A@END.bmp\x00'
>> + ##
>> + STRING_MS3D_REPLACE = 'use_ms3d_replace'
>> + STRING_ENCODING = "ascii" # wrong encoding (too limited), but there is an
>> UnicodeEncodeError issue, that prevent using the correct one for the moment
>> + ##STRING_ENCODING = "cp437" # US, wrong encoding and shows UnicodeEncodeError
>> + ##STRING_ENCODING = "cp858" # Europe + €, wrong encoding and shows UnicodeEncodeError
>> + ##STRING_ENCODING = "cp1252" # WIN EU, this would be the better codepage, but shows
>> UnicodeEncodeError, on print on system console and writing to file
>> + STRING_ERROR = STRING_MS3D_REPLACE
>> + ##STRING_ERROR = 'replace'
>> + ##STRING_ERROR = 'ignore'
>> + ##STRING_ERROR = 'surrogateescape'
>> + STRING_TERMINATION = b'\x00'
>> + STRING_REPLACE = u'_'
>> +
>> +
>> ###########################################################################
>> #
>> # min, max, default values
>> @@ -251,54 +270,41 @@
>> itemValue = value[i]
>> Ms3dIo.write_array(raw_io, itemWriter, count2, itemValue)
>>
>> +
>> @staticmethod
>> + def ms3d_replace(exc):
>> + """ http://www.python.org/dev/peps/pep-0293/ """
>> + if isinstance(exc, UnicodeEncodeError):
>> + return ((exc.end-exc.start)*Ms3dSpec.STRING_REPLACE, exc.end)
>> + elif isinstance(exc, UnicodeDecodeError):
>> + return (Ms3dSpec.STRING_REPLACE, exc.end)
>> + elif isinstance(exc, UnicodeTranslateError):
>> + return ((exc.end-exc.start)*Ms3dSpec.STRING_REPLACE, exc.end)
>> + else:
>> + raise TypeError("can't handle %s" % exc.__name__)
>> +
>> + @staticmethod
>> def read_string(raw_io, length):
>> """ read a string of a specific length from raw_io """
>> - value = []
>> - skip = False
>> - for i in range(length):
>> - buffer = raw_io.read(Ms3dIo.SIZE_SBYTE)
>> - if not buffer:
>> - raise EOFError()
>> - raw = (int)(unpack('<b', buffer)[0])
>> - if (raw >= 32) & (raw <= 255):
>> - pass
>> - else:
>> - if (raw == 0):
>> - raw = 0
>> - skip = True
>> - else:
>> - raw = 32
>> + buffer = raw_io.read(length)
>> + if not buffer:
>> + raise EOFError()
>> + eol = buffer.find(Ms3dSpec.STRING_TERMINATION)
>> + register_error(Ms3dSpec.STRING_MS3D_REPLACE, Ms3dIo.ms3d_replace)
>> + s = buffer[:eol].decode(encoding=Ms3dSpec.STRING_ENCODING,
>> errors=Ms3dSpec.STRING_ERROR)
>> + return s
>>
>> - c = chr(raw)
>> -
>> - if (not skip):
>> - value.append(c)
>> -
>> - finalValue = "".join(value)
>> - return finalValue
>> -
>> @staticmethod
>> def write_string(raw_io, length, value):
>> """ write a string of a specific length to raw_io """
>> - l = len(value)
>> - for i in range(length):
>> - if(i < l):
>> - c = value[i]
>> + register_error(Ms3dSpec.STRING_MS3D_REPLACE, Ms3dIo.ms3d_replace)
>> + buffer = value.encode(encoding=Ms3dSpec.STRING_ENCODING,
>> errors=Ms3dSpec.STRING_ERROR)
>> + if not buffer:
>> + buffer = bytes()
>> + raw_io.write(pack('<{}s'.format(length), buffer))
>> + return
>>
>> - if (isinstance(c, str)):
>> - c = c[0]
>> - raw = ord(c)
>> - elif (isinstance(c, int)):
>> - raw = c
>> - else:
>> - pass
>> - else:
>> - raw = 0
>>
>> - raw_io.write(pack('<b', (int)(raw % 256)))
>> -
>> -
>> ###############################################################################
>> #
>> # multi complex types
>>
>> _______________________________________________
>> Bf-extensions-cvs mailing list
>> Bf-extensions-cvs at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-extensions-cvs
>
> Not sure as to the details of this commit, but blender uses utf8
> encoding for all its own strings.
>
> Theres a problem where currently windows sys.stdout/stderr remains
> cp1252, but I wouldn't recommend to add anything cp1252 specific into
> your code.
>
> Report in tracker...
> http://projects.blender.org/tracker/index.php?func=detail&aid=31555&group_id=9&atid=498
>
> Python guys know about this...
> http://bugs.python.org/issue16129
>
> --
> - Campbell
> _______________________________________________
> Bf-extensions-cvs mailing list
> Bf-extensions-cvs at blender.org
> http://lists.blender.org/mailman/listinfo/bf-extensions-cvs
>
More information about the Bf-extensions-cvs
mailing list