[Bf-extensions-cvs] SVN commit: /data/svn/bf-extensions [4257] trunk/py/scripts/addons/ io_scene_ms3d: fix/workaround for character encoding ('ascii' is used for now, it should be 'cp1252' finally, but there are issues with printing on system console a
Campbell Barton
ideasman42 at gmail.com
Tue Feb 12 05:04:38 CET 2013
On Tue, Feb 12, 2013 at 8:27 AM, Beta Tester <alpha-beta-release at gmx.net> wrote:
> hi Campbell,
> i think i have to use that encoding, when i want to encode/decode namings of
> ms3d in a correct way, because ms3d natively use cp1252 as codepage (because
> it runs in a windows environment).
> so on import time i decode ms3d strings to blender strings (utf-8) and on
> export i encode blender strings back to ms3d compatible ones...
>
> but not yet, i use "ascii" instead "cp1252", for now (ascii is a subset of
> cp1252), because when i want to use cp1252, i get during windows system
> console output and later on export then i encode and write back to file an
> "UnicodeError" exception.
>
> https://projects.blender.org/tracker/index.php?func=detail&aid=34189
>
> or am i misunderstanding the meaning of decode / encode?
> bytes.decode('cp1252') brings a cp1252 encoded string to blenders utf-8.
> and str.encode('cp1252') brings blender strings back to cp1252 ones...
> is that correct... and when i use 'ascii' instead of 'cp1252' it does the
> same, but only this more limited codepage?
don't know about cp1252 but looks like you're going about this the
right way, your commit log made me think you were using windows
default encoding because its (incorrectly) default for our sys.stdout.
> ----- Original Message ----- From: "Campbell Barton" <ideasman42 at gmail.com>
> To: <bf-extensions-cvs at blender.org>
> Sent: Monday, February 11, 2013 3:17 AM
> Subject: Re: [Bf-extensions-cvs] SVN commit: /data/svn/bf-extensions [4257]
> trunk/py/scripts/addons/ io_scene_ms3d: fix/workaround for character
> encoding ('ascii' is used for now, it should be 'cp1252' finally, but there
> are issues with printing on system console and struct.pack or io
> .FileIO.write)
>
>
>> On Mon, Feb 11, 2013 at 3:46 AM, Alexander Nussbaumer
>> <alpha-beta-release at gmx.net> wrote:
>>>
>>> Revision: 4257
>>>
>>>
>>> http://projects.blender.org/scm/viewvc.php?view=rev&root=bf-extensions&revision=4257
>>> Author: beta-tester
>>> Date: 2013-02-10 16:46:54 +0000 (Sun, 10 Feb 2013)
>>> Log Message:
>>> -----------
>>> fix/workaround for character encoding ('ascii' is used for now, it should
>>> be 'cp1252' finally, but there are issues with printing on system console
>>> and struct.pack or io.FileIO.write)
>>>
>>> Modified Paths:
>>> --------------
>>> trunk/py/scripts/addons/io_scene_ms3d/__init__.py
>>> trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py
>>>
>>> Modified: trunk/py/scripts/addons/io_scene_ms3d/__init__.py
>>> ===================================================================
>>> --- trunk/py/scripts/addons/io_scene_ms3d/__init__.py 2013-02-10
>>> 14:51:42 UTC (rev 4256)
>>> +++ trunk/py/scripts/addons/io_scene_ms3d/__init__.py 2013-02-10
>>> 16:46:54 UTC (rev 4257)
>>> @@ -23,7 +23,7 @@
>>> 'description': "Import / Export MilkShape3D MS3D files"\
>>> " (conform with MilkShape3D v1.8.4)",
>>> 'author': "Alexander Nussbaumer",
>>> - 'version': (0, 95, 1),
>>> + 'version': (0, 95, 2),
>>> 'blender': (2, 65, 3),
>>> 'location': "File > Import & File > Export",
>>> 'warning': "",
>>>
>>> Modified: trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py
>>> ===================================================================
>>> --- trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py 2013-02-10
>>> 14:51:42 UTC (rev 4256)
>>> +++ trunk/py/scripts/addons/io_scene_ms3d/ms3d_spec.py 2013-02-10
>>> 16:46:54 UTC (rev 4257)
>>> @@ -37,6 +37,9 @@
>>> from sys import (
>>> exc_info,
>>> )
>>> +from codecs import (
>>> + register_error,
>>> + )
>>>
>>>
>>> ###############################################################################
>>> #
>>> @@ -97,6 +100,22 @@
>>> HEADER = "MS3D000000"
>>>
>>>
>>> + ## TEST_STR = 'START@€@µ@²@³@©@®@¶@ÿ@A at END.bmp'
>>> + ## TEST_RAW =
>>> b'START@\x80@\xb5@\xb2@\xb3@\xa9@\xae@\xb6@\xff at A@END.bmp\x00'
>>> + ##
>>> + STRING_MS3D_REPLACE = 'use_ms3d_replace'
>>> + STRING_ENCODING = "ascii" # wrong encoding (too limited), but there
>>> is an UnicodeEncodeError issue, that prevent using the correct one for the
>>> moment
>>> + ##STRING_ENCODING = "cp437" # US, wrong encoding and shows
>>> UnicodeEncodeError
>>> + ##STRING_ENCODING = "cp858" # Europe + €, wrong encoding and shows
>>> UnicodeEncodeError
>>> + ##STRING_ENCODING = "cp1252" # WIN EU, this would be the better
>>> codepage, but shows UnicodeEncodeError, on print on system console and
>>> writing to file
>>> + STRING_ERROR = STRING_MS3D_REPLACE
>>> + ##STRING_ERROR = 'replace'
>>> + ##STRING_ERROR = 'ignore'
>>> + ##STRING_ERROR = 'surrogateescape'
>>> + STRING_TERMINATION = b'\x00'
>>> + STRING_REPLACE = u'_'
>>> +
>>> +
>>>
>>> ###########################################################################
>>> #
>>> # min, max, default values
>>> @@ -251,54 +270,41 @@
>>> itemValue = value[i]
>>> Ms3dIo.write_array(raw_io, itemWriter, count2, itemValue)
>>>
>>> +
>>> @staticmethod
>>> + def ms3d_replace(exc):
>>> + """ http://www.python.org/dev/peps/pep-0293/ """
>>> + if isinstance(exc, UnicodeEncodeError):
>>> + return ((exc.end-exc.start)*Ms3dSpec.STRING_REPLACE,
>>> exc.end)
>>> + elif isinstance(exc, UnicodeDecodeError):
>>> + return (Ms3dSpec.STRING_REPLACE, exc.end)
>>> + elif isinstance(exc, UnicodeTranslateError):
>>> + return ((exc.end-exc.start)*Ms3dSpec.STRING_REPLACE,
>>> exc.end)
>>> + else:
>>> + raise TypeError("can't handle %s" % exc.__name__)
>>> +
>>> + @staticmethod
>>> def read_string(raw_io, length):
>>> """ read a string of a specific length from raw_io """
>>> - value = []
>>> - skip = False
>>> - for i in range(length):
>>> - buffer = raw_io.read(Ms3dIo.SIZE_SBYTE)
>>> - if not buffer:
>>> - raise EOFError()
>>> - raw = (int)(unpack('<b', buffer)[0])
>>> - if (raw >= 32) & (raw <= 255):
>>> - pass
>>> - else:
>>> - if (raw == 0):
>>> - raw = 0
>>> - skip = True
>>> - else:
>>> - raw = 32
>>> + buffer = raw_io.read(length)
>>> + if not buffer:
>>> + raise EOFError()
>>> + eol = buffer.find(Ms3dSpec.STRING_TERMINATION)
>>> + register_error(Ms3dSpec.STRING_MS3D_REPLACE,
>>> Ms3dIo.ms3d_replace)
>>> + s = buffer[:eol].decode(encoding=Ms3dSpec.STRING_ENCODING,
>>> errors=Ms3dSpec.STRING_ERROR)
>>> + return s
>>>
>>> - c = chr(raw)
>>> -
>>> - if (not skip):
>>> - value.append(c)
>>> -
>>> - finalValue = "".join(value)
>>> - return finalValue
>>> -
>>> @staticmethod
>>> def write_string(raw_io, length, value):
>>> """ write a string of a specific length to raw_io """
>>> - l = len(value)
>>> - for i in range(length):
>>> - if(i < l):
>>> - c = value[i]
>>> + register_error(Ms3dSpec.STRING_MS3D_REPLACE,
>>> Ms3dIo.ms3d_replace)
>>> + buffer = value.encode(encoding=Ms3dSpec.STRING_ENCODING,
>>> errors=Ms3dSpec.STRING_ERROR)
>>> + if not buffer:
>>> + buffer = bytes()
>>> + raw_io.write(pack('<{}s'.format(length), buffer))
>>> + return
>>>
>>> - if (isinstance(c, str)):
>>> - c = c[0]
>>> - raw = ord(c)
>>> - elif (isinstance(c, int)):
>>> - raw = c
>>> - else:
>>> - pass
>>> - else:
>>> - raw = 0
>>>
>>> - raw_io.write(pack('<b', (int)(raw % 256)))
>>> -
>>> -
>>>
>>> ###############################################################################
>>> #
>>> # multi complex types
>>>
>>> _______________________________________________
>>> Bf-extensions-cvs mailing list
>>> Bf-extensions-cvs at blender.org
>>> http://lists.blender.org/mailman/listinfo/bf-extensions-cvs
>>
>>
>> Not sure as to the details of this commit, but blender uses utf8
>> encoding for all its own strings.
>>
>> Theres a problem where currently windows sys.stdout/stderr remains
>> cp1252, but I wouldn't recommend to add anything cp1252 specific into
>> your code.
>>
>> Report in tracker...
>>
>> http://projects.blender.org/tracker/index.php?func=detail&aid=31555&group_id=9&atid=498
>>
>> Python guys know about this...
>> http://bugs.python.org/issue16129
>>
>> --
>> - Campbell
>> _______________________________________________
>> Bf-extensions-cvs mailing list
>> Bf-extensions-cvs at blender.org
>> http://lists.blender.org/mailman/listinfo/bf-extensions-cvs
>>
>
--
- Campbell
More information about the Bf-extensions-cvs
mailing list